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Executive  Summary 

The  Community  Nursing  Organization  (CNO)  Demonstration  is  an  innovative  approach  to  the 
provision  of  community  nursing  and  ambulatory  care  services  for  Medicare  beneficiaries.  Structured 
around  the  two  fundamental  concepts  of  nurse  case  management  and  capitated  payment,  CNOs 
attempt  to  promote  the  timely  and  appropriate  use  of  community  health  services  and  to  reduce  the  use 
of  costly  acute-care  services. 

In  order  to  explore  the  impact  of  this  model  of  care  delivery  on  cost  and  outcomes,  the  CNO 
Demonstration  was  created  by  the  Omnibus  Budget  Reconciliation  Act  (OBRA)  of  1987.  To  carry 
out  this  demonstration,  the  Centers  for  Medicare  and  Medicaid  Services  (formerly  the  Health  Care 
Financing  Administration)  entered  in  1993  into  cooperative  agreements  with  the  following  four 
eligible  organizations  to  serve  as  demonstration  providers: 

•  Carondelet  Health  Care,  Tucson,  AZ 

•  Carle  Clinic,  Urbana,  IL 

•  Living  at  Home/Block  Nurse  Program,  Minneapolis,  MN 

•  Visiting  Nurse  Service,  New  York,  NY 

OBRA  1987  also  mandated  an  evaluation  of  the  CNO  demonstration.  Abt  Associates  Inc.  was 
awarded  a  contract  to  provide  technical  assistance  to  the  sites  and  to  evaluate  the  effects  of  the 
demonstration  from  January  1994  until  July  of  1997. 

The  evaluation  design  permitted  especially  strong  results  because  applicants  to  the  CNOs  were 
randomized  to  treatment  (CNO)  or  control  (regular  Medicare)  groups.  Abt  Associates  Inc.  reported 
the  results  of  the  CNO  Evaluation  in  a  Second  Interim  Report  on  April  6,  1998  and  in  a  Final  Report 
on  April  13,  2000.  The  Medicare,  Medicaid  and  SCHIP  Benefits  Improvement  and  Protection  Act 
(BIPA)  of  2000  directed  the  Secretary  of  Health  and  Human  Services  to  investigate  whether  results 
from  the  CNO  demonstration  might  change  in  the  long  run,  since  gains  from  the  preventive  efforts  of 
the  CNOs  might  take  time  to  materialize.  At  the  same  time,  BIPA  reduced  the  inflation-adjusted 
capitation  rates  to  be  paid  to  the  sites  (by  15  percent  in  New  York  and  10  percent  at  the  other  sites) 
and  replaced  the  previously  mandated  evaluation  with  a  Preliminary  Report  due  no  later  than  July  I, 
2001  and  a  Final  Report  no  later  than  July  1,  2002.  This  Preliminary  Report  summarizes  the 
evaluation  of  the  extension  of  the  CNO  Demonstration  (known  as  Phase  H)  and  covers  the 
demonstration  period  from  January  of  1994  through  December  of  1999.  The  Final  Phase  O 
Evaluation  Report  will  add  one  more  year  to  the  study,  concluding  in  December  of  2000. 

The  evaluation  of  Phase  II  faces  a  major  problem  that  was  avoided  by  the  experimental  design  of 
Phase  I.  Because  the  randomization  requirement  was  maintained  only  intermittently  after  October 
1995,  and  control  group  members  were  permitted  to  enroll  in  the  CNO  after  October  1997,  analysis  of 
demonstration  enrollees  during  these  periods  must  address  the  possible  effects  of  selection  bias. 
Selection  bias  arises  when  individuals  decide  (self-select)  whether  to  participate  in  a  program  and 
when  some  of  the  factors  that  influence  their  decision  also  influence  the  outcomes  to  be  evaluated.  In 
the  CNO  Demonstration,  it  appeared  that  beneficiaries  who  chose  to  participate  consistently  used 
more  services  than  the  average  beneficiary  in  the  CNO  service  area.  Failure  to  account  for  this 
difference  would  confuse  the  selection  effect  with  the  treatment  effect,  overstating  the  latter. 
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Recognizing  the  selection  bias  problem,  this  evaluation  employed  two  complimentary  approaches: 


•  The  comparison  of  average  utilization  and  expenditures  exclusively  for  randomized 
beneficiaries  (Chapter  3),  and 

•  The  analysis  of  all  treatment  group  beneficiaries,  compared  to  a  reference  group  drawn  from 
the  Medicare  population  living  in  the  same  geographic  area  (Chapter  4). 

This  second  approach  required  more  complex  statistical  models  to  adjust  as  much  as  possible  for  the 
effects  of  selection  bias  (see  Appendix  A  for  methodological  details). 

With  respect  to  the  most  important  question,  whether  the  CNO  sites  achieved  budget  neutrality  for  the 
Medicare  program,  these  two  analyses  reached  the  same  conclusion:  they  did  not.  When  analysis  was 
restricted  to  randomized  beneficiaries,  Medicare  spending  per  person  per  month  was  higher  for 
members  of  the  treatment  group  than  for  members  of  the  control  group  and  these  differences  were 
statistically  significant  at  three  of  the  four  sites.  When  all  treatment  group  members  were  compared 
to  the  population-based  reference  group,  treatment  group  spending  per  person  per  month  was 
statistically  significantly  higher  at  all  four  sites,  and  these  differences  became  larger  and  more 
significant  over  time. 

To  amplify  the  importance  of  long-run  effects  and  minimize  the  influence  of  start-up  costs  on  these 
results,  the  project  team  restricted  the  sample  and  repeated  both  approaches.  For  the  first  approach, 
the  first  six  months  after  randomization  and  all  months  in  1994  were  excluded,  but  the  results  were 
very  similar.  For  the  second  approach,  all  beneficiaries  enrolled  less  than  six  months  were  excluded 
from  the  treatment  group,  producing  qualitatively  similar,  though  smaller,  estimated  differences 
between  groups.  This  slight  change  in  results  could  be  traced  to  the  disproportionate  exclusion  from 
the  sample  of  high-cost  beneficiaries,  who  were  more  likely  to  drop  out  of  the  CNO,  perhaps  because 
they  became  ineligible  due  to  institutionalization  or  because  they  found  case  management  to  be  too 
restrictive  (See  Table  3.3.5). 

Both  analytic  approaches  found  that  capitation  rates  for  CNO-covered  services  resulted  in  payments 
for  the  treatment  group  that  were  higher  than  payments  for  the  control  or  population  reference  groups. 
The  analysis  of  randomized  beneficiaries  indicated  that  this  was  the  only  source  of  elevated 
expenditures  for  the  treatment  group,  implying  that  a  reduction  in  capitation  rates  might  make  the 
CNO  budget  neutral.  By  contrast,  comparison  of  the  treatment  group  to  the  population  reference 
group  suggested  that  expenditures  for  non-CNO  services,  particularly  inpatient  hospitalization,  were 
also  higher  for  the  treatment  group,  implying  that  CNO  participation  induced  higher  utilization  of 
non-CNO  services  (only  at  LAH  and  VNS). 

We  will  further  investigate  this  phenomenon  as  the  evaluation  continues,  but  at  the  present  time  we 
hypothesize  that,  relative  to  the  population  reference  group,  both  treatment  and  control  groups 
became  more  likely  to  be  hospitalized  over  time  because  they  had  substantial  ongoing  contact  with 
the  same  provider.  For  example,  suppose  that  hospitalization  rates  were  going  down  in  Minnesota 
and  New  York  (perhaps  because  of  growth  in  managed  care),  but  this  trend  was  partially  offset 
among  beneficiaries  with  substantial  ongoing  contact  with  VNS  or  LAH  staff  who  may  have 
advocated  on  their  behalf  for  more  intensive  treatments.  This  practice  pattern  would  have  affected 
treatment  and  control  groups  equally  because  most  CNO  participants  were  recruited  from 
beneficiaries  who  were  already  involved  with  the  sites.   Consequently,  any  site-specific  practice 
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pattern  anomalies  would  not  be  observed  in  a  comparison  between  the  treatment  and  control  groups. 
Furthermore,  depending  on  the  way  in  which  a  CNO  benefit  might  be  incorporated  into  the  Medicare 
program,  this  effect  might  or  might  not  be  characterized  as  an  expected  effect  of  the  CNO.  In  other 
words,  if  the  new  CNO  benefit  were  aimed  exclusively  at  beneficiaries  who  already  had  substantial 
provider  contact,  there  might  be  no  expected  effect  on  hospital  expenditures.  On  the  other  hand,  if 
CNO  services  were  delivered  to  those  who  would  not  otherwise  have  had  such  contact,  some  growth 
in  hospital  expenditures  might  result. 

These  issues  will  be  further  investigated  in  the  fall  of  2001,  with  additional  data  from  calendar  year 
2000,  and  the  results  will  be  included  in  a  Final  Report  to  Congress.  In  addition  to  extended 
expenditure  and  utilization  analysis,  this  report  will  contain  results  from  analysis  of  mortality  and 
potentially  avoidable  hospitalizations,  two  measures  that  may  reflect  any  differences  in  quality  of  care 
received  by  CNO  participants. 

Questions  regarding  beneficiary  satisfaction  and  health  education  were  analyzed  in  the  Phase  I  Final 
Report.  Survey  responses  at  27  and  39  months  following  random  assignment  were  examined, 
indicating  no  superior  outcomes  at  27  months  and  some  improved  satisfaction  associated  with 
assignment  to  the  CNO  at  39  months.  Treatment  group  respondents  reported  greater  satisfaction  on 
two  measures:  nursing  care  and  participation  in  decisions  regarding  their  health  care.  These  changes 
may  be  attributable  to  CNO  performance;  however,  those  who  were  not  satisfied  may  have 
disenrolled  by  the  39-month  followup,  skewing  the  data.  BIPA  mandated  the  sites  to  conduct  a 
survey  of  beneficiary  satisfaction  during  Phase  II.  The  Final  Phase  II  Report  will  include  the  results 
of  this  survey. 
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1.    Introduction  to  the  Community  Nursing 
Organization  Evaluation 


The  Community  Nursing  Organization  (CNO)  Demonstration  is  an  innovative  approach  to  the 
provision  of  community  nursing  and  ambulatory  care  services  for  Medicare  beneficiaries.  Structured 
around  the  two  fundamental  concepts  of  nurse  case  management  and  capitated  payment,  CNOs 
attempt  to  promote  the  timely  and  appropriate  use  of  community  health  services  and  to  reduce  the  use 
of  costly  acute-care  services. 

The  impetus  for  developing  the  CNO  model  stemmed  from  limitations  in  traditional  fee-for-service 
Medicare.  Parts  A  and  B  of  Medicare  only  reimburse  care  that  is  ordered  by  a  physician  and  supplied 
by  certain  providers  under  certain  specified  conditions.  Medicare  generally  has  no  provision  for 
reimbursing  preventive  care,  health  promotion,  or  care  not  authorized  by  a  physician,  services  that 
might  lead  to  lower  medical  costs  and  improved  health  outcomes  for  Medicare  beneficiaries.  Since 
1985,  many  Medicare  HMOs  have  aimed  to  compensate  for  these  limitations  by  attempting  to  provide 
a  broader  and  more  flexible  array  of  services,  in  return  for  a  fixed  monthly  payment  for  each 
subscriber.  However,  many  Medicare  beneficiaries  are  reluctant  to  join  HMOs,  since  the 
organizations  typically  restrict  members'  choice  of  providers. 

The  CNO  concept  thus  provides  an  alternative  to  both  traditional  fee-for-service  Medicare  and 
Medicare  HMOs.  Like  HMOs,  CNOs  are  funded  by  flat  monthly  Medicare  payments  for  each 
enrolled  member  and  are  responsible  for  operating  within  that  budget,  but  can  exercise  substantial 
discretion  in  organizing  care  in  the  most  efficient  and  productive  way.  Since  only  a  limited  range  of 
services  is  covered  by  the  capitation  payment,  beneficiaries  are  still  able  to  choose  their  providers, 
notably  physicians,  hospitals  and  other  facilities  in  the  same  manner  as  all  other  Medicare  fee-for- 
service  beneficiaries.  However,  CNO  nurses  coordinate  the  provision  of  health  care  services  for  each 
enrollee  with  a  strong  focus  on  prevention  and  disease  management,  thus  attempting  to  avoid  higher 
future  health  care  costs  (Storfjell  1997,  Schraeder  1997,  Ethridge  1997). 


1.1.   Background  on  the  CNO  Demonstration 

In  order  to  explore  the  impact  of  this  model  of  care  delivery  on  cost  and  outcomes,  the  CNO 
Demonstration  was  created  by  the  Omnibus  Budget  Reconciliation  Act  (OBRA)  of  1987.  To  carry 
out  this  demonstration,  the  Centers  for  Medicare  and  Medicaid  Services  (formerly  the  Health  Care 
Financing  Administration)  in  1993  entered  into  cooperative  agreements  with  the  following  four 
eligible  organizations  to  serve  as  demonstration  providers: 

•  Carondelet  Health  Care,  Tucson,  AZ 

•  Carle  Clinic,  Urbana,  IL 

•  Living  at  Home/Block  Nurse  Program,  Minneapolis,  MN 

•  Visiting  Nurse  Service,  New  York,  NY 

OBRA  1987  also  mandated  an  evaluation  of  the  CNO  demonstration.  Abt  Associates  Inc.  was 
awarded  a  contract  to  provide  technical  assistance  to  the  sites  and  to  evaluate  the  effects  of  the 
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demonstration  from  January  1994  until  July  of  1997.  The  evaluation,  based  on  randomized 
assignment  of  applicants  to  either  CNO  participation  or  regular  Medicare  coverage,  was  aimed  at 
addressing  two  fundamental  questions: 

1.  Were  beneficiary  outcomes  such  as  health  status,  physical  functioning,  and  satisfaction  with 
health  care  improved  as  a  result  of  enrollment  in  the  CNO? 

2.  What  were  the  implications  of  the  CNO  demonstration  on  Medicare  program  costs? 

The  evaluation  design  permitted  especially  strong  answers  to  these  questions  because  applicants  to 
the  CNOs  were  randomized  to  treatment  (CNO)  or  control  (regular  Medicare)  groups.  Abt  Associates 
Inc.  reported  the  results  of  the  CNO  Evaluation  in  a  Second  Interim  Report  on  April  6,  1998  and  in  a 
Final  Report  on  April  13,  2000. 

Telephone  survey  responses  to  questions  designed  to  estimate  the  impact  of  the  intervention  on 
overall  physical  and  social  functioning  found  only  small  and  generally  insignificant  differences  in 
functional  status  between  treatment  and  control  groups  at  15,  27,  and  39  months  after  randomization. 
Total  Medicare  expenditures  were  found  to  be  significantly  higher  among  treatment  group 
beneficiaries  than  among  those  assigned  to  the  control  group.  This  result  held  regardless  of  whether 
the  treatment  group  was  defined  as  all  beneficiaries  randomly  assigned  to  treatment  (the  'intent-to- 
treat'  model)  or  was  defined  as  beneficiaries  assigned  to  treatment  and  actually  enrolled  in  the  CNO. 
These  results  were  in  close  agreement  with  those  of  other  studies  of  Medicare  risk  HMOs  carried  out 
during  the  1980s  and  1990s  (e.g.  Brown  et.  al.,  1993)  and  were  consistent  with  favorable  selection 
into  the  CNOs.1 

Questions  regarding  beneficiary  satisfaction  and  health  education  also  were  analyzed  in  the  Final 
Report.  Survey  responses  at  27  and  39  months  following  random  assignment  were  examined, 
indicating  no  superior  outcomes  at  27  months  and  some  improved  satisfaction  associated  with 
assignment  to  the  CNO  at  39  months.  Treatment  group  respondents  reported  greater  satisfaction  on 
two  measures:  nursing  care  and  participation  in  decisions  regarding  their  health  care.  These  changes 
may  be  attributable  to  CNO  performance;  however,  those  who  were  not  satisfied  may  have 
disenrolled  by  the  39-month  followup,  skewing  the  data. 

The  Medicare,  Medicaid  and  SCHJP  Benefits  Improvement  and  Protection  Act  (BIPA)  of  2000 
directed  the  Secretary  of  Health  and  Human  Services  to  investigate  whether  results  from  the  CNO 
demonstration  might  change  in  the  long  run,  since  gains  from  the  preventive  efforts  of  the  CNOs 
might  take  time  to  materialize.  At  the  same  time,  BIPA  reduced  the  inflation-adjusted  capitation  rates 
to  be  paid  to  the  sites  (by  15  percent  in  New  York  and  10  percent  at  the  other  sites)  and  replaced  the 
previously  mandated  evaluation  with  a  Preliminary  Report  due  no  later  than  July  1,  2001  and  a  Final 
Report  no  later  than  July  1 ,  2002.  This  Preliminary  Report  summarizes  the  evaluation  of  the  extension 
of  the  CNO  Demonstration  (known  as  Phase  EI)  and  covers  the  demonstration  period  from  January  of 
1994  through  December  of  1999.  BIPA  also  mandated  the  sites  to  conduct  a  survey  of  beneficiary 
satisfaction  during  Phase  n.  The  Final  Phase  U  Report  will  include  the  results  of  this  survey  as  well 
as  an  extension  of  the  analysis  in  the  Preliminary  report  by  adding  one  more  year  to  the  study, 
concluding  in  December  of  2000. 


Age-adjusted  mortality  rates  were  lower  for  CNO  applicants  than  for  eligible  non-applicants  residing  in  the 
same  localities. 
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1.1.1.        Participating  Sites 


Through  a  competitive  selection  process,  the  Centers  for  Medicare  and  Medicaid  Services  (CMS) 
chose  four  diverse  sites  to  set  up  CNOs  for  the  demonstration: 


•  Carle  Clinic  in  Urbana,  EL,  a  for-profit  private  physician  group  practice; 

•  Carondelet  Health  Care  in  Tucson,  AZ,  a  Catholic,  non-profit,  full-service  health  care 
corporation; 

•  Living  At  Home/Block  Nurse  Program  (LAH/BNP)  in  Minneapolis,  MN,  a  community- 
based  nursing  program  for  the  elderly,  in  partnership  with  HealthSpan,  the  largest  home 
health  agency  in  the  state;  and 

•  Visiting  Nurse  Service  of  New  York  (VNSNY)  in  New  York  City,  the  largest  non-profit 
Medicare  certified  home  health  agency  in  the  United  States. 

Each  site  had  considerable  freedom  in  how  it  chose  to  organize  itself.  As  long  as  the  mandatory 
services  were  provided  and  the  basic  OBRA  guidelines  were  followed,  sites  could  individually 
determine  the  most  efficient  and  productive  ways  to  serve  their  members.  Considerations  addressed 
by  each  CNO  included: 


Their  relationship  with  the  sponsoring  organization,  e.g.  how  would  the  sponsor  benefit 
from  the  CNO  demonstration; 
The  optimal  location  for  their  sites; 

How  to  recruit  members  (what  would  appeal  most  to  applicants  in  the  local  community?); 
How  to  maintain  the  financial  viability  of  the  project; 

How  to  define  the  roles  of  the  Primary  Nurse  Providers  (PNP)  and  other  staff  members; 
How  to  coordinate  the  provision  of  services  through  physicians  and  contracted  providers 
(since  the  CNO  itself  did  not  provide  physical  therapy,  home  health  care,  durable  medical 
equipment,  etc.); 

How  to  connect  enrollees  to  available  community  services; 

How  to  standardize  the  authorization  of  services  for  enrollees;  and 

How  to  encourage  the  continued  participation  of  enrollees  in  the  CNO. 


Since  the  sites  represent  diverse  locations,  and  clienteles,  they  have  responded  in  a  variety  of  ways  to 
these  considerations.  Below  each  CNO  site  is  briefly  discussed,  highlighting  the  manner  in  which  it 
chose  to  fulfill  the  OBRA  '87  mandate. 


Carle  Clinic  CNO 

Carle  Clinic,  the  sponsoring  organization  for  the  Carle  Clinic  CNO,  is  a  for-profit,  private  physician 
group  practice  with  a  large  ambulatory  nursing  component.  Serving  nearly  2,500  patients  daily,  the 
Carle  Clinic  organizations  act  as  the  regional  medical  center  for  the  primarily  rural  population  of 
Central  Illinois  and  Western  Indiana.  The  Carle  Clinic  system  is  designed  to  provide  primary  care 
through  a  network  of  clinics,  each  using  local  community  services  and  networking  with  local 
providers. 

By  mid-demonstration,  the  Carle  Clinic  CNO  was  operating  7  sites  that  served  predominantly  rural 
areas,  with  health  services  provided  by  13  PNPs.  Carle  Clinic  PNPs  provided  direct  care  and  case 
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management,  and  they  tended  to  be  paired  with  physicians  or  assigned  to  groups  of  physicians  who 
provided  a  wide  range  of  services,  including  services  to  non-CNO  enrollees.  PNPs  who  served 
higher-risk  enrollees  had  smaller,  more  specialized  caseloads  than  PNPs  serving  low-risk  clients.  By 
mid-demonstration,  seven  case  assistants  (CAs)  supported  the  PNPs  by  doing  administrative  work 
and  monitoring  low-risk  enrollees  by  telephone.  Because  of  the  rural  clientele  served  by  this  CNO, 
the  Carle  Clime  PNPs  relied  more  heavily  on  telephone  monitoring  of  their  patients  than  on  in-person 
visits;  there  were  also  fewer  opportunities  to  "drop  in"  here  than  at  the  other  CNOs.  Some  of  the 
contracted  providers  for  the  demonstration  were  affiliated  with  Carle  Clinic  while  others  were  not. 
Finally,  as  in  other  rural  areas,  managed  care  penetration  in  rural  Illinois  was  low  during  the 
demonstration,  making  beneficiaries  less  familiar  with  managed  care  practices  than  they  were  at  the 
other  demonstration  sites. 

Carondelet  Health  Care  CNO 

Carondelet  Health  Care  (CHC),  the  sponsoring  organization  for  the  Carondelet  CNO,  is  a  Catholic, 
non-profit,  full-service  health  care  corporation  that  has  operated  in  southern  Arizona  for  more  than 
100  years.  By  mid-demonstration,  the  Carondelet  CNO  had  21  community  sites  at  a  variety  of 
locations  including  senior  centers,  clinics,  mobile  home  parks,  and  housing  units.  All  of  these  sites 
were  accessible  to  both  CNO  and  non-CNO  enrollees.  The  CNO  utilized  some  of  CHC's  nurse  case 
managers,  community  health  centers,  outpatient  services,  and  its  home  health  agency.  Most  of  the 
contracted  providers  for  the  demonstration  were  affiliated  with  CHC,  although  there  were  no  formal 
relationships  between  PNPs  and  CHC  physicians. 

Two  distinct  types  of  nurses  worked  as  PNPs:  1)  nurse  case  managers,  usually  nurse  practitioners, 
who  traditionally  worked  with  higher  risk  individuals  who  were  hospitalized  or  home-bound;  and  2) 
nurse  partners,  usually  RNs,  who  worked  in  the  community  with  lower  risk  individuals.  If  the  low- 
risk  clients  moved  into  a  higher  risk  category,  they  were  assigned  to  a  nurse  case  manager. 

The  Tucson  area  in  which  the  Carondelet  CNO  operated  had  the  most  competitive  managed  care 
environment  of  the  four  sites.  Several  managed  care  programs  competed  directly  with  the  CNO.  The 
area  was  also  characterized  by  populations  of  retirees  who,  because  of  seasonal  migration  out  of  the 
service  area,  periodically  enrolled  and  disenrolled,  according  to  the  rules  of  the  CNO.  In  the  latter  part 
of  the  demonstration,  the  CNO  expanded  to  include  the  largely  Hispanic  populations  in  southern 
Arizona. 

Although  fully  operational  throughout  the  period  covered  by  this  report,  it  should  be  noted  that 
Carondelet  Health  Care  ceased  operations  as  a  CNO  demonstration  site  on  September  30,  2000. 

Living  at  Home/Block  Nurse  Program  CNO 

The  Living  At  Home/Block  Nurse  Program  Inc.  (LAH/BNP)  is  a  community-based  initiative  that  was 
first  piloted  in  St.  Paul,  MN  in  1982  and  has  grown  to  have  thirteen  programs  across  Minnesota.  The 
first  program  was  started  when  community  residents  organized  to  care  for  the  elderly  in  the 
community,  implementing  case  management  services  for  which  there  was  no  Medicare 
reimbursement.  To  set  up  the  CNO,  LAH/BNP  formed  a  contractual  relationship  with  HealthSpan, 


"Contracted  providers"  refers  to  any  agencies  authorized  by  the  CNO  to  provide  direct  health  services  to 
CNO  enrollees,  such  as  physical  therapy,  durable  medical  equipment,  home  health  care,  etc. 


Abt  Associates  Inc. 


Phase  II  Evaluation  of  CNO  Demonstration 


9 


the  largest  Medicare  certified  home  care  agency  in  the  state.  HealthSpan  provided  the  CNO  with 
nursing  staff,  financial  services,  and  home  care  services,  as  well  as  durable  medical  equipment.  PNPs 
had  to  forge  their  own  relationships  with  physicians  in  the  community. 

The  CNO  opened  two  rural  and  two  urban  sites,  all  of  which  served  CNO  enrollees  exclusively.  By 
mid-demonstration,  eight  PNPs  were  each  assigned  to  one  of  the  sites  to  provide  direct  care  and  case 
management  services.  Each  nurse  worked  with  a  mixture  of  high-  and  low-risk  individuals.  The 
CNO  incorporated  the  LAH/BNP  principles  of  self-governance  by  community  members,  including  an 
advisory  committee  and  an  emphasis  on  volunteers.  Each  CNO  site  employed  a  community 
coordinator  to  assist  with  non-health  services  and  coordinate  the  volunteers.  There  were  over  200 
volunteers  working  for  the  sites,  and  more  than  10  percent  of  them  are  CNO  enrollees. 

The  Minneapolis/St.  Paul  area  had  higher  managed  care  penetration  than  the  rest  of  Minnesota. 
However,  the  HMOs  in  this  area  tended  to  be  non-profit  entities  and  not  as  competitive  as  those  in  the 
Tucson  area.  HealthSpan  was  an  experienced  player  in  this  particular  market. 

Although  fully  operational  throughout  the  period  covered  by  this  report,  LAH  ceased  operations  as  a 
CNO  demonstration  site  on  November  30,  2000. 

Visiting  Nurse  Service  of  New  York  (VNSNY)  CNO 

Visiting  Nurse  Service  of  New  York  is  the  largest  non-profit  Medicare-Certified  Home  Health  Care 
Agency  in  the  nation,  providing  more  than  1 .2  million  professional  visits  annually  to  residents  of 
New  York  City.  By  mid-demonstration,  the  VNSNY  CNO  had  28  urban  sites,  all  in  Queens,  NY. 
Sites  were  located  in  various  organizations,  such  as  senior  centers  or  housing  units,  that  were 
accessible  to  both  CNO  and  non-CNO  enrollees.  Each  enrollee  was  assigned  to  one  of  the  ten  PNPs 
during  the  initial  assessment,  and  for  many  enrollees,  the  PNP  served  as  their  main  primary  care 
provider.  The  PNPs  carried  a  mixed  caseload  of  high-  and  low-risk  patients,  and  had  "office  hours" 
at  the  different  sites  during  which  enrollees  could  easily  drop  in. 

VNSNY  CNO  enrollees  tended  to  be  older  and  sicker  than  enrollees  at  other  sites.  Many  of  them 
lived  alone,  and  some  had  psychological  problems,  as  in  the  case  of  enrollees  who  were  Holocaust 
survivors.  This  CNO  therefore  had  a  heavier  emphasis  on  psychological  services  than  other  sites.  A 
member  services  assistant  at  the  central  office  would  identify  community  resources  for  enrollees,  but 
PNPs  had  the  main  responsibility  for  referring  enrollees  to  community  services.  The  VNSNY  CNO 
enrollees  were  reluctant  to  relinquish  access  to  services  that  they  believed  they  deserved  or  could 
obtain  elsewhere.  Physicians  and  other  contracted  providers,  such  as  physical  therapists,  tended  to 
respond  to  this  environment  by  being  independent  and  competitive,  presenting  some  challenges  for 
the  VNSNY  CNO. 

The  New  York  City  area  has  traditionally  been  resistant  to  managed  care,  in  comparison  to  other  parts 
of  the  state.  During  the  demonstration,  HMOs  in  the  New  York  City  area  became  increasingly 
interested  in  the  use  of  mid-level  and  non-traditional  providers  that  might  appeal  to  a  managed  care- 
resistant  population,  but  most  of  these  initiatives  appear  to  have  been  terminated  for  financial  reasons. 
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1.1.2.        Eligibility  and  Enrollment 

All  Medicare  beneficiaries  residing  in  catchment  areas  close  to  the  CNOs,  who  were  entitled  to 
benefits  under  Part  A  and  who  were  enrolled  in  Part  B  of  Medicare  were  eligible  to  enroll  in  the 
CNO,  with  the  following  exceptions: 

•  beneficiaries  enrolled  in  Medicare  risk  HMOs, 

•  beneficiaries  receiving  care  under  the  Medicare  hospice  benefit,  and 
beneficiaries  entitled  to  Medicare  under  the  End  Stage  Renal  Disease  (ESRD)  benefit. 


8 


Each  CNO  site  was  required  to  hold  at  least  one  open  enrollment  period  during  the  operational  Phase 
of  the  demonstration  and  to  accept  any  eligible  beneficiary  who  applied  for  membership.  Initially, 
those  accepted  into  the  demonstration  were  randomly  assigned  to  treatment  and  control  groups  for  the 
evaluation.  In  later  phases  of  the  demonstration,  direct  enrollment  into  the  treatment  arm  and 
switching  from  control  to  treatment  arm  became  possible. 

CNO  members  were  allowed  to  disenroll  at  the  end  of  a  calendar  month  for  any  reason.  No  enrollee 
could  be  forced  to  leave  the  CNO  due  to  high  service  use.  However,  under  the  following  conditions, 
a  CNO  was  required  to  disenroll  a  member.  These  conditions  were: 

•  failure  to  maintain  enrollment  in  Parts  A  and  B  of  Medicare, 

•  institutionalization  for  60  or  more  consecutive  days  (changed  to  30  days  in  1998), 

•  enrollment  in  a  Medicare  risk  HMO, 

•  use  of  the  Medicare  hospice  benefit, 

•  residence  outside  of  the  CNO  service  area  for  more  than  30  consecutive  days, 

•  persistent  use  of  out-of-plan  care  for  CNO  mandatory  services  while  enrolled  in  the 
CNO,  or 

•  refusal  of  mandatory  six-month  assessment.3 

Sites  began  randomization  and  enrollment  on  January  1,  1994  with  the  expectation  that  the 
demonstration  would  last  for  three  years.  In  1996,  CMS  extended  the  CNO  demonstration  and 
evaluation  for  an  additional  year.  The  Balanced  Budget  Act  of  1997  subsequently  granted  a  further 
two-year  extension  for  the  project.  Most  recently,  the  Balanced  Budget  Refinement  Act  of  1999 
granted  an  additional  two-year  extension,  authorizing  the  sites  to  continue  operating  until  December 
31,2001. 

1.1.3.        Covered  Services 

OBRA  1987  required  that  certain  services  be  provided  as  part  of  the  CNO  service  package.  These 
services  were  further  clarified  by  contracts  between  CMS  and  the  four  CNO  sites  to  include: 

•  Home  health  services  as  defined  in  42  CFR  409.40-409.42,  provided  by  qualified 
personnel  who  meet  the  qualifications  specified  in  42  CFR  484.4.  Home  health  services 
are  regular  Medicare  covered  home  health  agency  services  or  comparable  level  CNO 


In  some  cases  beginning  in  1998,  payment  rates  were  set  to  the  lowest  value  instead  of  disenrolling  the 
beneficiary. 
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services,  which  may  be  authorized  by  either  a  physician  or  a  CNO  nurse,  furnished  to 
home-bound  patients.  These  services  include: 

>■ 

-  part-time  or  intermittent  nursing  care  provided  by  or  under  the  supervision  of  a 
registered  professional  nurse; 

-  physical,  speech,  and  occupational  therapy; 

-  medical  social  services  supportive  plan  of  care;  and 

-  part-time  or  intermittent  services  of  a  home  health  aide  furnished  under  the 
supervision  of  a  registered  nurse. 

•  Medical  supplies,  appliances,  and  devices  as  defined  in  42  CFR  410.36,  including: 

-  surgical  dressings,  and  splints,  casts,  and  other  devices  used  for  reduction  of  fractures 
and  dislocations; 

-  prosthetic  devices,  other  than  dental,  that  replace  all  or  part  of  an  internal  body  organ, 
including  colostomy  bags  and  supplies  directly  related  to  colostomy  care;  and 

-  leg,  arm,  back,  and  neck  braces  and  artificial  legs,  arms,  and  eyes. 

•  Durable  medical  equipment  as  defined  in  42  CFR  410.38,  to  be  used  in  the  patient's 
home. 

•  Ambulance  services  as  defined  in  42  CFR  4 1 0.40,  when: 

-  medically  necessary  because  other  means  of  transportation  would  endanger  the 
beneficiary's  health; 

-  the  enrollee  is  not  a  hospital  inpatient;  and 

-  the  transportation  is  not  by  air  or  water. 

[Ambulance  service  was  removed  from  the  CNO  package  on  February  I,  1997.] 

•  Outpatient  physical  therapy  services  as  defined  in  42  CFR  410.60. 

•  Outpatient  speech  pathology  services  as  defined  in  42  CFR  410.62. 

•  Medical  supplies  (other  than  drugs  and  biologicals)  furnished  while  an  enrollee  is  under  a 
plan  of  care,  if  the  supplies  are  of  the  type  that  are  commonly  furnished  in  a  physician's 
office  or  clinic  and  are  commonly  furnished  either  without  charge  or  included  in  the 
physician's  or  clinic's  bill. 

•  Services  furnished  by  a  clinical  psychologist  who  meets  the  qualifications  specified  in  42 
CFR  410.71(d),  or  a  clinical  social  worker  as  defined  in  section  1861  (hh)  of  the  Social 
Security  Act,  as  well  as  services  and  supplies  furnished  as  an  incident  to  their  services. 

•  Part-time  or  intermittent  nursing  care  and  related  medical  supplies  (other  than  drugs  and 
biologicals)  furnished  by  a  registered  professional  or  licensed  practical  nurse  employed 
or  under  arrangement  with  a  Medicare  certified  rural  health  clinic. 
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•  Case  management  services  defined  as  services  which  assist  enrollees  in  gaining  access  to 
and  coordinating/approving  utilization  of  needed  medical,  social,  educational  and  other 
services.  In  the  CNO,  this  service  must  include  providing  an  in-persoh  assessment  and 
updating  the  patient's  care  plan  every  six  months.  This  service  also  includes  coordinating 
these  services  with  other  providers  and  monitoring  the  enrollee's  progress  towards  the 
achievement  of  objectives  specified  in  the  patient's  CNO  plan. 


Thus,  expenditures  were  classified  as  CNO  or  non-CNO  expenditures  for  the  purposes  of  the 
evaluation,  as  follows: 


Table  1.3.1 

Allocation  of  Medicare  expenditures  to  CNO  and  non-CNO  services 


CNO  Service  Package  ("in-bundle") 

CNO  capitation  payments 

CNO  case  management  payments 

Home  health  care  (6  disciplines) 

Outpatient  physical  therapy 

Durable  medical  equipment 

Prosthetics/orthotics 

Supplies 


Non-CNO  services  ("out-of-bundle") 

Inpatient  hospital  (short  and  long  stay) 
Hospital  outpatient 
Skilled  nursing  facility 
Hospice 

Physician  office  visits 
Physician  other 

Part  B  other  (lab,  ancillary,  other) 


Note  that  enrolled  members  of  the  treatment  group  should  have  little  or  no  in-bundle  costs  except  for  capitation  and  case  management 
payments. 


Source:  Abt  Associates  coding  algorithms 


1.1.4.        Capitation  and  Case  Mix  Adjustment 

Each  CNO  received  a  monthly  payment  for  each  enrolled  member.  Payment  amounts  were  based  on 
the  local  average  annual  per  capita  cost  for  Medicare-covered  services  that  were  part  of  the  CNO's 
package.  These  rates  in  turn  were  adjusted  for  case  mix  as  directed  by  OBRA.  In  all  sites,  payments 
were  adjusted  for  age,  sex,  and  number  of  Medicare-covered  home  health  visits  in  the  previous  six 
months.  In  three  of  the  sites  (AZ,  MN  and  NY)  payments,  were  further  adjusted  for  the  number  of 
limitations  in  activities  of  daily  living  (ADLs)  experienced  by  the  enrollee.  This  resulted  in  a  total  of 
39  different  payment  levels,  also  called  payment  cells,  for  those  three  sites.  Payments  to  the  Carle 
Clinic  (IL)  site  were  not  adjusted  for  ADL  limitations  and  were  based  on  13  payment  cells. 
Following  each  6-month  reassessment,  enrollees  were  reassigned  to  the  payment  cell  appropriate  for 
their  current  age,  home  health  utilization,  and  (in  three  sites)  number  of  ADL  limitations.  The  total 
payment  that  a  CNO  site  received  per  member  per  month  consisted  of  this  risk  adjusted  capitation  fee 
plus  a  case  management  fee  that  was  the  same  for  each  enrollee  and  each  site. 
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1.1.5.  Case  Management 

Aside  from  the  requirement  that  every  CNO  member  be  evaluated  in  person  at  siximonth  intervals, 
each  of  the  CNO  sites  was  free  to  define  and  configure  the  process  of  case  management  in  the  way  it 
judged  to  be  most  beneficial  to  the  member  and  efficient  for  the  organization.  Methods  of 
assessment,  resources  devoted  to  planning  and  monitoring,  as  well  as  the  number  of  members  whose 
care  was  actively  managed,  therefore,  differ  from  site  to  site.  Although  the  benefits  and  cost 
effectiveness  of  case  management  for  the  frail  or  chronically  ill  are  fairly  well  established  (Cohen, 
1991),  the  value  of  case  management  in  the  broader  population  of  the  "generally  well  elderly" 
remains  unknown.  Because  the  demonstration  has  only  four  sites  and  because  the  case  management 
intervention  was  not  experimentally  varied  across  sites  or  individuals,  the  evaluation  was  unable  to 
distinguish  the  distinct  effects  of  capitation  and  case  management  on  beneficiary  outcomes, 
utilization,  or  cost. 

1 .1 .6.  Recruitment  and  Intake 

Each  site  developed  its  own  strategy  for  marketing  and  recruitment  of  eligible  beneficiaries.  All  sites 
relied  on  physician  referrals,  direct  mail,  and  word  of  mouth.  Some  sites  also  used  brochures,  fliers, 
group  presentations,  television  and  newspaper  advertising,  and  telemarketing  efforts.  Because  the 
demonstration  was  conducted  as  an  experiment,  with  random  assignment  to  treatment  or  control 
groups,  it  was  important  that  beneficiaries  who  expressed  interest  in  the  program  understood  that 
there  was  a  certain  probability  that  they  would  be  assigned  to  a  control  group  and  not  be  enrolled  in 
the  CNO.  Sites  were  therefore  required  to  secure  informed  consent  from  each  applicant.  The  consent 
document  informed  applicants 

1 .  that  the  CNO  was  a  temporary  demonstration  project, 

2.  that,  if  enrolled,  they  must  agree  to  receive  all  care  in  the  CNO  service  package  only  from  the 
CNO, 

3 .  that  they  would  be  enrolled  in  the  CNO  only  if  assigned  to  the  treatment  group,  and 

4.  that  they  would  be  contacted  by  Abt  Associates  for  telephone  interviews  at  one-year 
intervals. 

After  securing  informed  consent  from  the  applicant,  a  CNO  staff  person  conducted  a  baseline 
interview  with  the  applicant.  The  interview  elicited  information  on  health,  mental  status,  functional 
limitations,  health  risk,  demographic  characteristics,  and  attitudes  toward  health  providers  and 
satisfaction  with  care.  Applicants  were  randomized  after  the  interview.  Applicants  assigned  to  the 
control  group  were  thanked  for  their  participation  and  informed  that  they  could  not  receive  services 
from  the  CNO.  If  necessary,  applicants  assigned  to  the  treatment  group  were  further  assessed  to 
facilitate  care  planning  and  case  management,  and  were  enrolled  in  the  CNO. 


1 .2.  Design  of  the  CNO  Evaluation 

The  evaluation  of  the  CNO  Demonstration  was  designed  so  that  the  impact  of  the  intervention  could 
be  readily  measured.  Implementation  of  any  novel  approach  to  health  care  delivery  is,  however,  a 
dynamic  process  where  theoretical  design  concepts  may  sometimes  be  altered  to  accommodate 
operational  constraints.  CMS,  the  sites,  and  the  evaluation  contractor  collaborated  in  an  effort  to 
balance  issues  related  to  implementation  with  issues  related  to  evaluating  the  effect  of  CNOs  on 
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outcomes  and  cost.  The  compromises  that  were  necessary  and  their  implications  for  the  evaluation 
are  discussed  below. 

1.2.1.  Experimental  Design 

In  order  to  develop  the  most  precise  estimates  possible  of  the  impacts  of  the  CNO  intervention,  the 
demonstration  was  structured  as  a  social  experiment  in  which  individuals  were  randomized  to  either 
CNO  participation  (the  treatment  group)  or  to  continue  to  receive  their  traditional  Medicare  benefits 
(the  control  group).  However,  given  that  participation  in  the  CNO  was  voluntary,  the  decision  to 
apply  was  likely  to  be  influenced  by  hard-to-measure  factors  that  also  influence  health  outcomes  and 
cost.  The  subset  of  the  Medicare  population  that  wished  to  participate  in  the  CNO  was  likely  to  differ 
from  those  who  had  no  interest  in  joining  the  CNO.  Thus,  the  only  way  to  create  a  valid  comparison 
strategy  was  to  do  so  after  the  decision  to  participate  in  the  evaluation  had  been  made  so  that  only 
those  who  wished  to  participate  could  be  compared.  Applicants  were  randomly  assigned  to  treatment 
or  control  status  after  the  decision  to  apply  had  been  made,  a  consent  statement  had  been  signed,  and 
collection  of  baseline  data  had  occurred. 

1.2.2.  Implementation  of  Random  Assignment 

To  accommodate  the  program's  need  to  build  up  enrollment  quickly,  two  applicants  were  assigned  to 
the  treatment  group  for  every  applicant  assigned  to  the  control  group.  The  fact  that  the  control  group 
was  smaller  than  the  treatment  group  reduced  the  statistical  power  of  the  evaluation,  increasing  the 
size  of  the  minimum  impact  that  could  be  detected  reliably.  In  determining  what  proportion  of 
applicants  to  enroll  in  the  CNOs,  the  size  of  the  impact  that  could  be  detected  (and  therefore  the 
threshold  for  being  considered  a  significant  impact)  was  balanced  against  the  sites'  need  to  recruit 
more  participants.4 

To  avoid  potential  bias  on  the  part  of  the  CNO  site  staff,  who  conducted  the  baseline  assessments, 
baseline  data  on  the  CNO  applicants  were  collected  before  the  applicants  were  randomized.  Thus 
while  they  were  performing  the  baseline  assessment,  the  assessors  did  not  know  whether  the  applicant 
would  in  fact  be  able  to  enroll  in  the  CNO.  In  order  to  facilitate  and  control  the  randomization 
process,  Abt  Associates  developed  a  centralized  CNO  Random  Assignment  System  (CNORAS) 
maintained  at  Abt  Associates'  offices  in  Cambridge,  MA.  After  the  baseline  was  performed,  site  staff 
were  able  to  dial  into  this  database  via  laptop  computer  and  modem  and  to  enter  basic  data  on  each 
new  applicant.  The  system  then  assigned  each  applicant  to  the  treatment  or  control  group  and  gave 
them  a  unique  identifier.  Site  staff  could  copy  down  the  identifier  and  the  assignment  and  enter  it  in 
site  records.  If  an  enrollee  was  already  in  the  database,  the  system  indicated  his  or  her  existing 
identifier  and  treatment/control  assignment  status. 

Members  of  the  same  household  who  applied  to  the  CNO  were  automatically  assigned  to  the  same 
treatment/control  status.  This  was  done  to  avoid  problems  in  service  delivery  within  the  household 


4     For  example,  it  was  estimated  that  an  assignment  ratio  of  2: 1  meant  that  an  8  percent  reduction  in  the  rate 
of  inpatient  admissions  could  be  detected  with  statistical  power  of  .7 1  (at  a  .  10  significance  level), 
assuming  total  enrollment  of  4,800  (3,200  in  the  treatment  group  and  1,600  in  the  control  group). 
Allocating  to  treatment  and  control  groups  using  a  1:1  ratio  would  have  allowed  a  smaller  impact  to  be 
detected  with  comparable  power,  but  would  have  resulted  only  2,400  treatment  participants,  unless  sites 
recruited  a  larger  total  number  of  applicants  to  yield  the  same  number  of  enrollees  (3,200). 
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and  the  likelihood  of  control  group  members  benefiting  from  CNO  services  provided  to  treatment 
group  members  in  the  same  household.  To  facilitate  this  assignment,  site  staff  identified  the  potential 
eligible  members  of  each  applicant's  household;  these  were  termed  Qualified  Household  Members 
(QHMs).  Data  on  all  QHMs  were  entered  into  the  CNORAS,  even  if  they  were  not  applying  to  the 
CNO.  QHMs  who  later  decided  to  apply  would  hence  automatically  be  assigned  to  the  proper  group. 
This  led  to  a  slight  increase  in  the  ratio  of  treatments  to  controls,  since  QHMs  of  control  group 
members  generally  did  not  apply  to  the  CNO. 

1.2.3.        Special  Situations 

The  original  specifications  for  the  implementation  of  random  assignment  called  for  the  following 
sequence.  First,  the  beneficiary  would  be  recruited  by  the  site  and  sign  an  informed  consent  form 
accepting  participation  in  random  assignment.  Then  collection  of  baseline  assessment  and  other  data 
for  the  evaluation  would  occur.  Once  baseline  data  collection  was  complete,  the  randomization 
assignment  would  be  requested  from  Abt  Associates.  Control  group  members  would  be  informed  of 
their  status  and  have  no  further  contact  with  the  CNO.  Treatment  group  members  would  be  enrolled, 
receive  a  clinical  assessment,  and  begin  to  receive  CNO  services.  Unfortunately  for  the  evaluation, 
this  sequence  frequently  had  to  be  altered  in  practice.  The  most  common  exceptions  are  described 
below. 

•  Randomization  before  baseline  assessment:  Three  of  the  four  sites  lacked  laptop 
computers  that  would  allow  staff  to  call  in  to  CNORAS  from  applicants'  homes.  At  the 
outset,  site  staff  protested  that  it  was  awkward  and  inefficient  to  conduct  a  baseline 
assessment,  leave  and  obtain  the  random  assignment,  and  return  at  some  later  date  to 
perform  a  clinical  assessment  and  develop  a  care  plan  for  treatment  group  members. 
Eventually,  it  was  agreed  that  CNO  office  staff  could  call  in  to  CNORAS  for  cases  that 
were  to  be  assessed  that  day,  obtain  the  assignments,  conceal  them  in  an  envelope,  and 
provide  them  to  the  assessment  nurse.  Once  the  baseline  assessment  was  completed,  the 
nurse  could  reveal  the  random  assignment.  If  the  applicant  were  assigned  to  the  control 
group,  the  nurse  would  thank  them  and  leave;  if  assigned  to  the  treatment  group,  she 
could  continue  with  the  enrollment  and  care  planning  process. 

•  Randomization  without  baseline  assessment  or  enrollment:  There  were  some  situations 
where  beneficiaries  were  randomly  assigned  but  never  received  a  baseline  assessment,  or 
were  assigned  to  treatment  status  but  never  enrolled  in  the  CNO.  This  included  cases 
where  the  beneficiary  changed  his/her  mind  about  participation  after  being  randomized; 
where  the  site  assigned  the  beneficiary  to  treatment  or  control  status  before  s/he  had 
agreed  to  participate;  where  the  beneficiary  was  determined  to  be  ineligible  for  the  CNO 
after  being  randomized;  and  where  the  beneficiary  died  before  baseline  or  enrollment. 
These  cases  were  relatively  rare,  but  they  do  occupy  "slots"  in  the  CNORAS,  and  may 
therefore  cause  the  analysis  samples  to  depart  from  the  2: 1  ratio.  Treatments  and  controls 
who  received  no  baseline  assessment  did  not  receive  follow-up  assessments  from  Abt 
Associates. 


5 


The  system  allowed  site  staff  to  link  each  applicant  with  one  QHM,  which  covered  the  vast  majority  of 
situations  encountered.  Occasionally,  an  applicant  had  multiple  QHMs.  These  were  reported  to  Abt  on  a 
case-by-case  basis,  and  Abt  staff  established  the  link  in  the  CNORAS  manually. 
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•  Contaminated  Controls:  In  several  instances,  beneficiaries  who  were  randomly  assigned 
to  the  control  group  were  inadvertently  enrolled  in  the  CNO  and  received  the  same 
services  as  a  member  of  the  treatment  group.  The  intent-to-treat  design  of  the  evaluation 
mandated  that  these  cases  were  nonetheless  analyzed  as  controls.  This  was  a  relatively 
minor  problem  before  October  1997.  After  this  date,  however,  the  sites  enrolled  control 
group  members  in  substantial  numbers. 

•  Hiatus  in  randomization  to  the  control  group:  The  CNO  Evaluation  was  originally 
scheduled  to  end  on  December  31,  1995.  Starting  October  1,  1995,  all  new  applicants 
were  "randomized"  to  the  treatment  group,  since  no  follow-up  assessments  allowing 
comparisons  between  treatments  and  controls  would  have  been  performed  on  applicants 
randomized  after  that  date.  In  early  1996  CMS  modified  the  original  contract  allowing 
the  evaluation  to  continue  for  another  year.  At  that  point  in  time,  it  was  decided  that  the 
pool  of  control  group  members  was  already  sufficiently  large  and  that  randomizing  a 
small  number  of  new  controls  would  contribute  little  to  the  analysis.  Throughout  1996, 
all  applicants  were  assigned  to  the  treatment  group.  However,  when  the  demonstration 
was  again  extended  for  two  more  years,  the  randomization  of  new  applicants  to  both 
treatment  and  control  groups  in  a  2:1  ratio  was  resumed.    Overall,  the  hiatus  in 
randomization  to  the  control  group  lasted  from  October  1995  through  December  1996. 
As  a  result,  all  1,144  CNO  applicants  during  that  time  period  were  enrolled  as  treatments, 
and  the  overall  ratio  of  treatments  to  controls  became  greater  than  2:1.  This  period  is 
sometimes  referred  to  as  Wave  2  of  the  evaluation,  with  the  initial  phase  with  intact 
randomization  being  labeled  Wave  I.  When  it  had  been  decided  to  continue  the 
demonstration  for  two  additional  years,  randomization  in  a  2:1  ratio  was  resumed  from 
January  1,  1997  until  October  2,  1997  (Wave  3).  After  October  3,  1997,  sites  again  were 
allowed  to  enroll  applicants  without  randomization  (Wave  4). 

1.2.4.        Comparison  Strategies 

This  evaluation  employs  three  comparison  strategies.  The  first  two  are  comparisons  of  mean 
utilization  and  expenditures  that  rely  on  the  experimental  design  of  the  evaluation.  These  strategies 
are  directly  comparable  to  analyses  performed  for  the  Phase  I  Evaluation.  The  third  strategy  was 
added  to  address  the  fact  that  significant  numbers  of  beneficiaries  enrolled  in  the  CNOs  during 
periods  when  randomization  did  not  occur.  These  enrollees  have  no  appropriate  control  group,  so  an 
alternative  reference  group  had  to  be  constructed  and  comparisons  had  to  be  adjusted  for  known 
differences  between  the  treatment  group  and  the  reference  group. 

The  primary  analytic  strategy  for  evaluation  of  CNO  effects  has  been  the  "intent  to  treat"  approach 
commonly  employed  in  the  analysis  of  clinical  trials  (Lachin,  2000).  This  method  compares 
Medicare  expenditures  for  beneficiaries  assigned  to  the  treatment  group  with  expenditures  for  those 
assigned  to  the  control  group  regardless  of  whether  or  not  those  assigned  to  treatment  remained  in  the 
CNO  for  the  entire  follow-up  period.  This  comparison  strategy  is  typically  selected  for  randomized 
studies  in  order  to  emulate  real  world  conditions  and  to  thus  make  results  generalizable.  Were  a  CNO 
program  to  become  part  of  the  Medicare  benefit,  it  is  most  unlikely  that  beneficiaries  would  be 
compelled  to  remain  in  the  CNO  once  they  had  joined.  Rather,  beneficiaries  would  be  permitted  to 
leave  the  program  and  some  would  do  so,  just  as  they  did  in  the  demonstration.  Given  this 
expectation,  it  would  be  unrealistic  to  compare  only  months  in  which  beneficiaries  were  actually 
enrolled.  The  treatment/control  difference  thus  answers  the  question,  "For  beneficiaries  likely  to 
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enroll  in  a  CNO  program,  what  is  the  average  monthly  saving  to  Medicare  of  giving  them  the  option 
to  do  so?"  In  other  words,  the  treatment/control  contrast  is  meant  to  estimate  the  effect  of  assignment 
to  the  treatment  (CNO)  group  on  the  trajectory  of  Medicare  expenditures  regardless  of  future  events 
that  might  lead  beneficiaries  to  leave  or  become  ineligible  for  the  CNO. 

A  second  comparison  strategy  was  added  upon  request  by  the  CNO  sites,  which  contrasts  the  control 
group  with  only  the  months  in  which  a  beneficiary  was  actually  enrolled  in  a  CNO.  The  rationale 
behind  this  enrollee/comparison  group  contrast  was  to  provide  an  upper  bound  of  what  effect  CNO 
enrollment  could  have  had,  if  all  treatment  group  members  had  remained  enrolled  in  a  CNO. 
However,  the  risk  of  differential  attrition  causes  this  contrast  to  be  problematic.  Participants 
randomized  to  treatment  who  experience  deteriorating  health  during  the  study  might  choose  to 
disenroll  disproportionately  because  they  felt  constrained  in  their  choice  of  providers.  Under  this 
assumption,  only  a  relatively  healthy  subgroup  remains  enrolled  in  the  CNO,  whereas  the  control 
group's  average  health  status  does  not  change.  The  rernaining  enrollees  will  be  on  average  healthier 
than  the  control  group  members,  and  will  therefore  have  lower  expenditures  and  better  outcomes.  As 
one  cannot  adequately  account  for  those  disenrollment  decisions,  it  is  not  possible  to  quantify  whether 
differences  between  the  treatment  and  control  groups  are  caused  by  a  treatment  effect  or  by 
differential  attrition.  Thus,  if  this  contrast  is  reported,  it  will  overstate  the  beneficial  effect  of  the 
CNOs  on  cost  and  outcomes. 

A  final  comparison  strategy  contrasted  CNO  enrollees  to  Medicare  beneficiaries  who  lived  in  the 
same  geographic  area  but  never  applied  to  a  CNO.  Since  the  initial  evaluation  focused  on  enrollees 
from  Wave  1  of  the  demonstration,  during  which  the  randomization  design  was  fully  intact,  a  simple 
comparison  of  means  was  sufficient  to  identify  the  effect  of  the  intervention.  However,  this  project 
analyzes  enrollees  from  Waves  2  and  4,  during  which  study  participants  were  enrolled  without  a 
corresponding  control  group.  In  addition,  as  all  participants  randomized  to  control  status  were  given 
the  option  to  switch  to  treatment  after  October  3,  1997,  some  of  the  original  controls  are  now 
'contaminated'  by  virtue  of  having  later  joined  the  CNO.  Hence  estimates  computed  from  post- 1997 
data  will  inevitably  be  less  protected  against  bias  due  to  self-selection  than  will  estimates  computed 
with  1994-1997  data.  Our  tabulations  of  the  CNO  Enrollment  File  indicate  that  approximately  18 
percent  of  applicants  (697  out  of  3,784)  originally  assigned  to  the  control  group  subsequently  enrolled 
in  the  CNO.  This  share  is  low  enough  that  the  original  gains  from  randomization  might  not  have  been 
fully  lost  through  its  subsequent  abandonment.  However,  the  need  arises  to  test  this  hypothesis  by 
constructing  a  reference  group  from  Medicare  beneficiaries  who  never  applied  to  a  CNO.  The 
selection  of  this  reference  group  is  discussed  in  Chapter  2,  and  details  on  our  non-experimental 
comparison  strategy  are  presented  in  Appendix  A. 

1 .3.   Data  Sources 

For  this  report  the  project  team  analyzed  data  from  the  following  sources: 

•  Medicare  enrollment  and  claims  files  from  CMS 

•  CNO  Enrollment  and  Payment  Files  maintained  by  CMS  staff  overseeing  the 
demonstration 

•  Hierarchical  Coexisting  Conditions  (HCC)  files  constructed  by  Fu  Associates 
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services,  which  may  be  authorized  by  either  a  physician  or  a  CNO  nurse,  furnished  to 
home-bound  patients.  These  services  include: 

-  part-time  or  intermittent  nursing  care  provided  by  or  under  the  supervision  of  a 
registered  professional  nurse; 

-  physical,  speech,  and  occupational  therapy; 

-  medical  social  services  supportive  plan  of  care;  and 

-  part-time  or  intermittent  services  of  a  home  health  aide  furnished  under  the 
supervision  of  a  registered  nurse. 

•  Medical  supplies,  appliances,  and  devices  as  defined  in  42  CFR  410.36,  including: 

-  surgical  dressings,  and  splints,  casts,  and  other  devices  used  for  reduction  of  fractures 
and  dislocations; 

-  prosthetic  devices,  other  than  dental,  that  replace  all  or  part  of  an  internal  body  organ, 
including  colostomy  bags  and  supplies  directly  related  to  colostomy  care;  and 

-  leg,  arm,  back,  and  neck  braces  and  artificial  legs,  arms,  and  eyes. 

•  Durable  medical  equipment  as  defined  in  42  CFR  410.38,  to  be  used  in  the  patient's 
home. 

•  Ambulance  services  as  defined  in  42  CFR  410.40,  when: 

-  medically  necessary  because  other  means  of  transportation  would  endanger  the 
beneficiary's  health; 

-  the  enrollee  is  not  a  hospital  inpatient;  and 

-  the  transportation  is  not  by  air  or  water. 

[Ambulance  service  was  removed  from  the  CNO  package  on  February  1,  1997.] 

•  Outpatient  physical  therapy  services  as  defined  in  42  CFR  410.60. 

•  Outpatient  speech  pathology  services  as  defined  in  42  CFR  4 10.62. 

•  Medical  supplies  (other  than  drugs  and  biologicals)  furnished  while  an  enrollee  is  under  a 
plan  of  care,  if  the  supplies  are  of  the  type  that  are  commonly  furnished  in  a  physician's 
office  or  clinic  and  are  commonly  furnished  either  without  charge  or  included  in  the 
physician's  or  clinic's  bill. 

•  Services  furnished  by  a  clinical  psychologist  who  meets  the  qualifications  specified  in  42 
CFR  410.71(d),  or  a  clinical  social  worker  as  defined  in  section  1861  (hh)  of  the  Social 
Security  Act,  as  well  as  services  and  supplies  furnished  as  an  incident  to  their  services. 

•  Part-time  or  intermittent  nursing  care  and  related  medical  supplies  (other  than  drugs  and 
biologicals)  furnished  by  a  registered  professional  or  licensed  practical  nurse  employed 
or  under  arrangement  with  a  Medicare  certified  rural  health  clinic. 
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•  Case  management  services  defined  as  services  which  assist  enrollees  in  gaining  access  to 
and  coordinating/approving  utilization  of  needed  medical,  social,  educational  and  other 
services.  In  the  CNO,  this  service  must  include  providing  an  in-persoh  assessment  and 
updating  the  patient's  care  plan  every  six  months.  This  service  also  includes  coordinating 
these  services  with  other  providers  and  monitoring  the  enrollee's  progress  towards  the 
achievement  of  objectives  specified  in  the  patient's  CNO  plan. 

Thus,  expenditures  were  classified  as  CNO  or  non-CNO  expenditures  for  the  purposes  of  the 
evaluation,  as  follows: 


Table  1.3.1 

Allocation  of  Medicare  expenditures  to  CNO  and  non-CNO  services 


CNO  Service  Package  ("in-bundle") 

CNO  capitation  payments 

CNO  case  management  payments 

Home  health  care  (6  disciplines) 

Outpatient  physical  therapy 

Durable  medical  equipment 

Prosthetics/orthotics 

Supplies 


Non-CNO  services  ("out-of-bundle") 

Inpatient  hospital  (short  and  long  stay) 
Hospital  outpatient 
Skilled  nursing  facility 
Hospice 

Physician  office  visits 
Physician  other 

Part  B  other  (lab,  ancillary,  other) 


Note  that  enrolled  members  of  the  treatment  group  should  have  little  or  no  in-bundle  costs  except  for  capitation  and  case  management 
payments. 


Source:  Abt  Associates  coding  algorithms 


1 . 1 .4.        Capitation  and  Case  M  ix  Adj  ustment 

Each  CNO  received  a  monthly  payment  for  each  enrolled  member.  Payment  amounts  were  based  on 
the  local  average  annual  per  capita  cost  for  Medicare-covered  services  that  were  part  of  the  CNO's 
package.  These  rates  in  turn  were  adjusted  for  case  mix  as  directed  by  OBRA.  In  all  sites,  payments 
were  adjusted  for  age,  sex,  and  number  of  Medicare-covered  home  health  visits  in  the  previous  six 
months.  In  three  of  the  sites  (AZ,  MN  and  NY)  payments  were  further  adjusted  for  the  number  of 
limitations  in  activities  of  daily  living  (ADLs)  experienced  by  the  enrollee.  This  resulted  in  a  total  of 
39  different  payment  levels,  also  called  payment  cells,  for  those  three  sites.  Payments  to  the  Carle 
Clinic  (IL)  site  were  not  adjusted  for  ADL  limitations  and  were  based  on  13  payment  cells. 
Following  each  6-month  reassessment,  enrollees  were  reassigned  to  the  payment  cell  appropriate  for 
their  current  age,  home  health  utilization,  and  (in  three  sites)  number  of  ADL  limitations.  The  total 
payment  that  a  CNO  site  received  per  member  per  month  consisted  of  this  risk  adjusted  capitation  fee 
plus  a  case  management  fee  that  was  the  same  for  each  enrollee  and  each  site. 
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1.1.5.  Case  Management 

Aside  from  the  requirement  that  every  CNO  member  be  evaluated  in  person  at  six^month  intervals, 
each  of  the  CNO  sites  was  free  to  define  and  configure  the  process  of  case  management  in  the  way  it 
judged  to  be  most  beneficial  to  the  member  and  efficient  for  the  organization.  Methods  of 
assessment,  resources  devoted  to  planning  and  monitoring,  as  well  as  the  number  of  members  whose 
care  was  actively  managed,  therefore,  differ  from  site  to  site.  Although  the  benefits  and  cost 
effectiveness  of  case  management  for  the  frail  or  chronically  ill  are  fairly  well  established  (Cohen, 
1991),  the  value  of  case  management  in  the  broader  population  of  the  "generally  well  elderly" 
remains  unknown.  Because  the  demonstration  has  only  four  sites  and  because  the  case  management 
intervention  was  not  experimentally  varied  across  sites  or  individuals,  the  evaluation  was  unable  to 
distinguish  the  distinct  effects  of  capitation  and  case  management  on  beneficiary  outcomes, 
utilization,  or  cost. 

1.1.6.  Recruitment  and  Intake 

Each  site  developed  its  own  strategy  for  marketing  and  recruitment  of  eligible  beneficiaries.  All  sites 
relied  on  physician  referrals,  direct  mail,  and  word  of  mouth.  Some  sites  also  used  brochures,  fliers, 
group  presentations,  television  and  newspaper  advertising,  and  telemarketing  efforts.  Because  the 
demonstration  was  conducted  as  an  experiment,  with  random  assignment  to  treatment  or  control 
groups,  it  was  important  that  beneficiaries  who  expressed  interest  in  the  program  understood  that 
there  was  a  certain  probability  that  they  would  be  assigned  to  a  control  group  and  not  be  enrolled  in 
the  CNO.  Sites  were  therefore  required  to  secure  informed  consent  from  each  applicant.  The  consent 
document  informed  applicants 

1 .  that  the  CNO  was  a  temporary  demonstration  project, 

2.  that,  if  enrolled,  they  must  agree  to  receive  all  care  in  the  CNO  service  package  only  from  the 
CNO, 

3.  that  they  would  be  enrolled  in  the  CNO  only  if  assigned  to  the  treatment  group,  and 

4.  that  they  would  be  contacted  by  Abt  Associates  for  telephone  interviews  at  one-year 
intervals. 

After  securing  informed  consent  from  the  applicant,  a  CNO  staff  person  conducted  a  baseline 
interview  with  the  applicant.  The  interview  elicited  information  on  health,  mental  status,  functional 
limitations,  health  risk,  demographic  characteristics,  and  attitudes  toward  health  providers  and 
satisfaction  with  care.  Applicants  were  randomized  after  the  interview.  Applicants  assigned  to  the 
control  group  were  thanked  for  their  participation  and  informed  that  they  could  not  receive  services 
from  the  CNO.  If  necessary,  applicants  assigned  to  the  treatment  group  were  further  assessed  to 
facilitate  care  planning  and  case  management,  and  were  enrolled  in  the  CNO. 

1 .2.   Design  of  the  CNO  Evaluation 

The  evaluation  of  the  CNO  Demonstration  was  designed  so  that  the  impact  of  the  intervention  could 
be  readily  measured.  Implementation  of  any  novel  approach  to  health  care  delivery  is,  however,  a 
dynamic  process  where  theoretical  design  concepts  may  sometimes  be  altered  to  accommodate 
operational  constraints.  CMS,  the  sites,  and  the  evaluation  contractor  collaborated  in  an  effort  to 
balance  issues  related  to  implementation  with  issues  related  to  evaluating  the  effect  of  CNOs  on 
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outcomes  and  cost.  The  compromises  that  were  necessary  and  their  implications  for  the  evaluation 
are  discussed  below. 

1.2.1.  Experimental  Design 

In  order  to  develop  the  most  precise  estimates  possible  of  the  impacts  of  the  CNO  intervention,  the 
demonstration  was  structured  as  a  social  experiment  in  which  individuals  were  randomized  to  either 
CNO  participation  (the  treatment  group)  or  to  continue  to  receive  their  traditional  Medicare  benefits 
(the  control  group).  However,  given  that  participation  in  the  CNO  was  voluntary,  the  decision  to 
apply  was  likely  to  be  influenced  by  hard-to-measure  factors  that  also  influence  health  outcomes  and 
cost.  The  subset  of  the  Medicare  population  that  wished  to  participate  in  the  CNO  was  likely  to  differ 
from  those  who  had  no  interest  in  joining  the  CNO.  Thus,  the  only  way  to  create  a  valid  comparison 
strategy  was  to  do  so  after  the  decision  to  participate  in  the  evaluation  had  been  made  so  that  only 
those  who  wished  to  participate  could  be  compared.  Applicants  were  randomly  assigned  to  treatment 
or  control  status  after  the  decision  to  apply  had  been  made,  a  consent  statement  had  been  signed,  and 
collection  of  baseline  data  had  occurred. 

1.2.2.  Implementation  of  Random  Assignment 

To  accommodate  the  program's  need  to  build  up  enrollment  quickly,  two  applicants  were  assigned  to 
the  treatment  group  for  every  applicant  assigned  to  the  control  group.  The  fact  that  the  control  group 
was  smaller  than  the  treatment  group  reduced  the  statistical  power  of  the  evaluation,  increasing  the 
size  of  the  minimum  impact  that  could  be  detected  reliably.  In  determining  what  proportion  of 
applicants  to  enroll  in  the  CNOs,  the  size  of  the  impact  that  could  be  detected  (and  therefore  the 
threshold  for  being  considered  a  significant  impact)  was  balanced  against  the  sites'  need  to  recruit 
more  participants.4 

To  avoid  potential  bias  on  the  part  of  the  CNO  site  staff,  who  conducted  the  baseline  assessments, 
baseline  data  on  the  CNO  applicants  were  collected  before  the  applicants  were  randomized.  Thus 
while  they  were  performing  the  baseline  assessment,  the  assessors  did  not  know  whether  the  applicant 
would  in  fact  be  able  to  enroll  in  the  CNO.  In  order  to  facilitate  and  control  the  randomization 
process,  Abt  Associates  developed  a  centralized  CNO  Random  Assignment  System  (CNORAS) 
maintained  at  Abt  Associates'  offices  in  Cambridge,  MA.  After  the  baseline  was  performed,  site  staff 
were  able  to  dial  into  this  database  via  laptop  computer  and  modem  and  to  enter  basic  data  on  each 
new  applicant.  The  system  then  assigned  each  applicant  to  the  treatment  or  control  group  and  gave 
them  a  unique  identifier.  Site  staff  could  copy  down  the  identifier  and  the  assignment  and  enter  it  in 
site  records.  If  an  enrollee  was  already  in  the  database,  the  system  indicated  his  or  her  existing 
identifier  and  treatment/control  assignment  status. 

Members  of  the  same  household  who  applied  to  the  CNO  were  automatically  assigned  to  the  same 
treatment/control  status.  This  was  done  to  avoid  problems  in  service  delivery  within  the  household 


For  example,  it  was  estimated  that  an  assignment  ratio  of  2: 1  meant  that  an  8  percent  reduction  in  the  rate 
of  inpatient  admissions  could  be  detected  with  statistical  power  of  .7 1  (at  a  .  10  significance  level), 
assuming  total  enrollment  of  4,800  (3,200  in  the  treatment  group  and  1,600  in  the  control  group). 
Allocating  to  treatment  and  control  groups  using  a  1 : 1  ratio  would  have  allowed  a  smaller  impact  to  be 
detected  with  comparable  power,  but  would  have  resulted  only  2,400  treatment  participants,  unless  sites 
recruited  a  larger  total  number  of  applicants  to  yield  the  same  number  of  enrollees  (3,200). 
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and  the  likelihood  of  control  group  members  benefiting  from  CNO  services  provided  to  treatment 
group  members  in  the  same  household.  To  facilitate  this  assignment,  site  staff  identified  the  potential 
eligible  members  of  each  applicant's  household;  these  were  termed  Qualified  Household  Members 
(QHMs).  Data  on  all  QHMs  were  entered  into  the  CNORAS,  even  if  they  were  not  applying  to  the 
CNO.  QHMs  who  later  decided  to  apply  would  hence  automatically  be  assigned  to  the  proper  group.5 
This  led  to  a  slight  increase  in  the  ratio  of  treatments  to  controls,  since  QHMs  of  control  group 
members  generally  did  not  apply  to  the  CNO. 

1.2.3.        Special  Situations 

The  original  specifications  for  the  implementation  of  random  assignment  called  for  the  following 
sequence.  First,  the  beneficiary  would  be  recruited  by  the  site  and  sign  an  informed  consent  form 
accepting  participation  in  random  assignment.  Then  collection  of  baseline  assessment  and  other  data 
for  the  evaluation  would  occur.  Once  baseline  data  collection  was  complete,  the  randomization 
assignment  would  be  requested  from  Abt  Associates.  Control  group  members  would  be  informed  of 
their  status  and  have  no  further  contact  with  the  CNO.  Treatment  group  members  would  be  enrolled, 
receive  a  clinical  assessment,  and  begin  to  receive  CNO  services.  Unfortunately  for  the  evaluation, 
this  sequence  frequently  had  to  be  altered  in  practice.  The  most  common  exceptions  are  described 
below. 

•  Randomization  before  baseline  assessment:  Three  of  the  four  sites  lacked  laptop 
computers  that  would  allow  staff  to  call  in  to  CNORAS  from  applicants'  homes.  At  the 
outset,  site  staff  protested  that  it  was  awkward  and  inefficient  to  conduct  a  baseline 
assessment,  leave  and  obtain  the  random  assignment,  and  return  at  some  later  date  to 
perform  a  clinical  assessment  and  develop  a  care  plan  for  treatment  group  members. 
Eventually,  it  was  agreed  that  CNO  office  staff  could  call  in  to  CNORAS  for  cases  that 
were  to  be  assessed  that  day,  obtain  the  assignments,  conceal  them  in  an  envelope,  and 
provide  them  to  the  assessment  nurse.  Once  the  baseline  assessment  was  completed,  the 
nurse  could  reveal  the  random  assignment.  If  the  applicant  were  assigned  to  the  control 
group,  the  nurse  would  thank  them  and  leave;  if  assigned  to  the  treatment  group,  she 
could  continue  with  the  enrollment  and  care  planning  process. 

•  Randomization  without  baseline  assessment  or  enrollment:  There  were  some  situations 
where  beneficiaries  were  randomly  assigned  but  never  received  a  baseline  assessment,  or 
were  assigned  to  treatment  status  but  never  enrolled  in  the  CNO.  This  included  cases 
where  the  beneficiary  changed  his/her  mind  about  participation  after  being  randomized; 
where  the  site  assigned  the  beneficiary  to  treatment  or  control  status  before  s/he  had 
agreed  to  participate;  where  the  beneficiary  was  determined  to  be  ineligible  for  the  CNO 
after  being  randomized;  and  where  the  beneficiary  died  before  baseline  or  enrollment. 
These  cases  were  relatively  rare,  but  they  do  occupy  "slots"  in  the  CNORAS,  and  may 
therefore  cause  the  analysis  samples  to  depart  from  the  2: 1  ratio.  Treatments  and  controls 
who  received  no  baseline  assessment  did  not  receive  follow-up  assessments  from  Abt 
Associates. 


5     The  system  allowed  site  staff  to  link  each  applicant  with  one  QHM,  which  covered  the  vast  majority  of 
situations  encountered.  Occasionally,  an  applicant  had  multiple  QHMs.  These  were  reported  to  Abt  on  a 
case-by-case  basis,  and  Abt  staff  established  the  link  in  the  CNORAS  manually. 
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•  Contaminated  Controls:  In  several  instances,  beneficiaries  who  were  randomly  assigned 
to  the  control  group  were  inadvertently  enrolled  in  the  CNO  and  received  the  same 
services  as  a  member  of  the  treatment  group.  The  intent-to-treat  design  of  the  evaluation 
mandated  that  these  cases  were  nonetheless  analyzed  as  controls.  This  was  a  relatively 
minor  problem  before  October  1997.  After  this  date,  however,  the  sites  enrolled  control 
group  members  in  substantial  numbers. 

•  Hiatus  in  randomization  to  the  control  group:  The  CNO  Evaluation  was  originally 
scheduled  to  end  on  December  31,  1995.  Starting  October  1,  1995,  all  new  applicants 
were  "randomized"  to  the  treatment  group,  since  no  follow-up  assessments  allowing 
comparisons  between  treatments  and  controls  would  have  been  performed  on  applicants 
randomized  after  that  date.  In  early  1996  CMS  modified  the  original  contract  allowing 
the  evaluation  to  continue  for  another  year.  At  that  point  in  time,  it  was  decided  that  the 
pool  of  control  group  members  was  already  sufficiently  large  and  that  randomizing  a 
small  number  of  new  controls  would  contribute  little  to  the  analysis.  Throughout  1996, 
all  applicants  were  assigned  to  the  treatment  group.  However,  when  the  demonstration 
was  again  extended  for  two  more  years,  the  randomization  of  new  applicants  to  both 
treatment  and  control  groups  in  a  2:1  ratio  was  resumed.  Overall,  the  hiatus  in 
randomization  to  the  control  group  lasted  from  October  1995  through  December  1996. 
As  a  result,  all  1,144  CNO  applicants  during  that  time  period  were  enrolled  as  treatments, 
and  the  overall  ratio  of  treatments  to  controls  became  greater  than  2:1.  This  period  is 
sometimes  referred  to  as  Wave  2  of  the  evaluation,  with  the  initial  phase  with  intact 
randomization  being  labeled  Wave  1.  When  it  had  been  decided  to  continue  the 
demonstration  for  two  additional  years,  randomization  in  a  2: 1  ratio  was  resumed  from 
January  1,  1997  until  October  2,  1997  (Wave  3).  After  October  3,  1997,  sites  again  were 
allowed  to  enroll  applicants  without  randomization  (Wave  4). 

1.2.4.         Comparison  Strategies 

This  evaluation  employs  three  comparison  strategies.  The  first  two  are  comparisons  of  mean 
utilization  and  expenditures  that  rely  on  the  experimental  design  of  the  evaluation.  These  strategies 
are  directly  comparable  to  analyses  performed  for  the  Phase  I  Evaluation.  The  third  strategy  was 
added  to  address  the  fact  that  significant  numbers  of  beneficiaries  enrolled  in  the  CNOs  during 
periods  when  randomization  did  not  occur.  These  enrollees  have  no  appropriate  control  group,  so  an 
alternative  reference  group  had  to  be  constructed  and  comparisons  had  to  be  adjusted  for  known 
differences  between  the  treatment  group  and  the  reference  group. 

The  primary  analytic  strategy  for  evaluation  of  CNO  effects  has  been  the  "intent  to  treat"  approach 
commonly  employed  in  the  analysis  of  clinical  trials  (Lachin,  2000).  This  method  compares 
Medicare  expenditures  for  beneficiaries  assigned  to  the  treatment  group  with  expenditures  for  those 
assigned  to  the  control  group  regardless  of  whether  or  not  those  assigned  to  treatment  remained  in  the 
CNO  for  the  entire  follow-up  period.  This  comparison  strategy  is  typically  selected  for  randomized 
studies  in  order  to  emulate  real  world  conditions  and  to  thus  make  results  generalizable.  Were  a  CNO 
program  to  become  part  of  the  Medicare  benefit,  it  is  most  unlikely  that  beneficiaries  would  be 
compelled  to  remain  in  the  CNO  once  they  had  joined.  Rather,  beneficiaries  would  be  permitted  to 
leave  the  program  and  some  would  do  so,  just  as  they  did  in  the  demonstration.  Given  this 
expectation,  it  would  be  unrealistic  to  compare  only  months  in  which  beneficiaries  were  actually 
enrolled.  The  treatment/control  difference  thus  answers  the  question,  "For  beneficiaries  likely  to 
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enroll  in  a  CNO  program,  what  is  the  average  monthly  saving  to  Medicare  of  giving  them  the  option 
to  do  so?"  In  other  words,  the  treatment/control  contrast  is  meant  to  estimate  the  effect  of  assignment 
to  the  treatment  (CNO)  group  on  the  trajectory  of  Medicare  expenditures  regardless  of  future  events 
that  might  lead  beneficiaries  to  leave  or  become  ineligible  for  the  CNO. 

A  second  comparison  strategy  was  added  upon  request  by  the  CNO  sites,  which  contrasts  the  control 
group  with  only  the  months  in  which  a  beneficiary  was  actually  enrolled  in  a  CNO.  The  rationale 
behind  this  enrollee/comparison  group  contrast  was  to  provide  an  upper  bound  of  what  effect  CNO 
enrollment  could  have  had,  if  all  treatment  group  members  had  remained  enrolled  in  a  CNO. 
However,  the  risk  of  differential  attrition  causes  this  contrast  to  be  problematic.  Participants 
randomized  to  treatment  who  experience  deteriorating  health  during  the  study  might  choose  to 
disenroll  disproportionately  because  they  felt  constrained  in  their  choice  of  providers.  Under  this 
assumption,  only  a  relatively  healthy  subgroup  remains  enrolled  in  the  CNO,  whereas  the  control 
group's  average  health  status  does  not  change.  The  remaining  enrollees  will  be  on  average  healthier 
than  the  control  group  members,  and  will  therefore  have  lower  expenditures  and  better  outcomes.  As 
one  cannot  adequately  account  for  those  disenrollment  decisions,  it  is  not  possible  to  quantify  whether 
differences  between  the  treatment  and  control  groups  are  caused  by  a  treatment  effect  or  by 
differential  attrition.  Thus,  if  this  contrast  is  reported,  it  will  overstate  the  beneficial  effect  of  the 
CNOs  on  cost  and  outcomes. 

A  final  comparison  strategy  contrasted  CNO  enrollees  to  Medicare  beneficiaries  who  lived  in  the 
same  geographic  area  but  never  applied  to  a  CNO.  Since  the  initial  evaluation  focused  on  enrollees 
from  Wave  1  of  the  demonstration,  during  which  the  randomization  design  was  fully  intact,  a  simple 
comparison  of  means  was  sufficient  to  identify  the  effect  of  the  intervention.  However,  this  project 
analyzes  enrollees  from  Waves  2  and  4,  during  which  study  participants  were  enrolled  without  a 
corresponding  control  group.  In  addition,  as  all  participants  randomized  to  control  status  were  given 
the  option  to  switch  to  treatment  after  October  3,  1997,  some  of  the  original  controls  are  now 
'contaminated'  by  virtue  of  having  later  joined  the  CNO.  Hence  estimates  computed  from  post- 1997 
data  will  inevitably  be  less  protected  against  bias  due  to  self-selection  than  will  estimates  computed 
with  1994-1997  data.  Our  tabulations  of  the  CNO  Enrollment  File  indicate  that  approximately  18 
percent  of  applicants  (697  out  of  3,784)  originally  assigned  to  the  control  group  subsequently  enrolled 
in  the  CNO.  This  share  is  low  enough  that  the  original  gains  from  randomization  might  not  have  been 
fully  lost  through  its  subsequent  abandonment.  However,  the  need  arises  to  test  this  hypothesis  by 
constructing  a  reference  group  from  Medicare  beneficiaries  who  never  applied  to  a  CNO.  The 
selection  of  this  reference  group  is  discussed  in  Chapter  2,  and  details  on  our  non-experimental 
comparison  strategy  are  presented  in  Appendix  A. 

1 .3.   Data  Sources 

For  this  report  the  project  team  analyzed  data  from  the  following  sources: 

•  Medicare  enrollment  and  claims  files  from  CMS 

•  CNO  Enrollment  and  Payment  Files  maintained  by  CMS  staff  overseeing  the 
demonstration 

•  Hierarchical  Coexisting  Conditions  (HCC)  files  constructed  by  Fu  Associates 
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1.3.1. 


Medicare  Enrollment  and  Claims  Files 


Medicare  service  utilization  and  eligibility  information  was  obtained  from  databases  maintained  by 
CMS.  The  National  Claims  History  Database  (NCH)  contains  Part  A  and  Part  B  claims  records, 
including  line  item  information  on  all  services  provided,  for  all  claims  since  October  1,  1990.  To 
analyze  each  beneficiary's  utilization  of  Medicare  services,  we  collected  the  following  information 
from  the  Inpatient,  Skilled  Nursing  Facility  (SNF),  Outpatient,  Home  Health  Agency  (HHA), 
Hospice,  and  Physician/Supplier  claims  files  for  all  randomized  beneficiaries: 

•  Beneficiary  identification  numbers  (Medicare  health  insurance  claim  numbers  (HICN)) 

•  Provider  identification  numbers 

•  Dates  of  service 

•  Type  of  claims  (inpatient,  SNF,  outpatient,  HHA,  hospice,  physician/supplier,  etc.) 

•  Units  of  service 

•  Submitted  charges 

•  Allowed  charges 

•  Reimbursement  amount 

•  Coinsurance  and  deductible  amounts 

•  Type  of  service  codes 

•  Place  of  service  codes 

•  Diagnosis  codes 

•  Procedure  codes 

Claims  records  were  collected  for  the  years  1993  to  1999  for  all  demonstration  participants  and  the 
population-based  reference  group. 

For  members  of  the  control  and  population  reference  groups,  Medicare  claims  files  provided 
information  on  cost  of  all  services,  regardless  of  whether  or  not  they  were  part  of  the  CNO  package. 
Since  the  CNO  package  services  were  paid  prospectively  by  the  capitation  fee  for  the  treatment  group, 
the  CNOs  assumed  full  financial  risk  and  did  not  bill  Medicare  for  these  services.  The  Medicare 
claims  files,  therefore,  should  only  have  contained  information  on  cost  of  services  that  were  not  part  of 
the  CNO  package,  such  as  physician  or  hospital  services.  However,  we  discovered  that  some 
members  of  the  treatment  group  had  obtained  CNO  package  services  outside  of  their  CNO.  This  was 
possible  because  of  the  absence  of  a  lockout  mechanism  that  would  have  prevented  payments  from  out 
of  plan  use  of  services.  Instead,  CMS  staff  followed  such  out-of-plan  use  and  recouped  the  capitation 
payment  for  every  month  in  which  out-of-plan  reimbursements  exceeded  $120  ($100  in  1995). 

The  CMS  Enrollment  Database  (EDB)  contains  demographic  data  elements  as  well  as  the  entitlement 
status  of  all  Medicare  beneficiaries.  The  project  team  used  this  information  to  determine  beneficiaries' 
eligibility  for  this  demonstration.  The  EDB  provided  the  following  types  of  information  about 
Medicare  enrollees: 

•  Identification  numbers  (Medicare  HICN) 

•  Demographic  information  (date  of  birth,  sex,  race,  state,  county,  zip  code) 

•  Date  of  death 

•  Medicare  Part  A  entitlement  and/or  Part  B  enrollment  and  termination  dates 

•  End  Stage  Renal  Disease  (ESRD)  entitlement 
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Disability  entitlement 

Group  Health  Organization  start  dates,  end  dates,  and  lock-in  codes 
Hospice  start  and  end  dates 


1.3.2.  CNO  Enrollment  and  Payment  Files 

These  files  were  maintained  by  CMS  to  determine  CNO  eligibility  both  at  the  time  of  enrollment  and 
on  a  continuous  basis  during  the  demonstration.  They  also  provided  information  on  how  many 
individuals  were  actually  enrolled  at  each  CNO  site,  by  month.  In  addition,  these  files  contained 
capitation  rates,  group  cell  categories  to  which  enrollees  were  assigned,  and  corresponding  assessment 
dates.  The  enrollment  files  allowed  CMS  to  keep  accurate  eligibility  records,  and  were  necessary  for 
CMS  to  determine  capitation  payments  for  the  CNOs  each  month.  The  same  CMS  project  staff 
maintained  files  reflecting  adjustments  to  the  capitation  payments  resulting  from  out-of-plan  use  as 
discussed  above. 

1.3.3.  HCC  Files 

As  mentioned  earlier,  because  of  the  departure  from  a  strict  randomized  design  there  is  no  control 
group  for  enrollees  from  Waves  2  and  4  of  the  demonstration.  Thus,  to  be  able  to  analyze  the  effect  of 
the  intervention  on  all  participants,  a  reference  group  had  to  be  constructed.  Because  of  potential 
differences  in  average  health  status  between  the  treatment  group  and  the  reference  group,  a  risk 
adjustment  technique  was  needed  to  separate  the  potential  demonstration  effect  from  these 
background  differences.  Hierarchical  Coexisting  Conditions  (HCCs)  are  a  prominent  method  for  this 
kind  of  risk  adjustment,  as  they  are  groups  based  on  ICD  9  diagnoses  that  are  predictive  of  future 
utilization  of  health  care  (Ellis  et.  al.,  1996).  HCC  scores  for  both  demonstration  participants  and  the 
population-based  reference  group  were  calculated  by  Fu  Associates,  under  a  direct  contract  with 
CMS,  and  made  available  to  the  project  team. 

1.4.   Incentives  and  Expectations 

Experiments  with  delegation  of  healthcare  decision-making  and  authority  delivered  under  a  capitation 
arrangement  usually  aim  at  familiar  goals  —  either  enhancing  health  and  well-being  without 
substantially  increasing  cost,  or  reducing  cost  with  no  measurable  sacrifice  in  health,  functioning,  or 
satisfaction.  This  naturally  leads  us  to  ask  what  scope  of  action  was  available  to  the  CNOs  to  effect 
improvements  in  cost  and  outcomes. 

The  CNO  demonstration  altered  the  provision  of  ambulatory  care  to  the  treatment  group  in  two  ways. 
First,  the  CNOs  assumed  full  financial  risk  for  all  care  in  the  CNO  service  package,  in  return  for  a 
monthly  capitation  payment  for  each  enrollee.  Second,  the  CNOs  provided  nurse  case  management  to 
all  enrollees,  including  in-person  assessments  for  all  members  at  six-month  intervals.  These 
alterations  gave  rise  to  three  mechanisms  by  which  CNOs  could  alter  directly  the  manner  in  which 
resources  were  used  to  maintain  and  improve  the  health  and  functioning  of  enrollees. 

•  The  CNO  was  accorded  much  greater  discretion  in  the  provision  of  Medicare-covered 
services.  Hence,  in  principle,  the  individual  needs  of  an  enrollee  could  be  accorded 
greater  importance  than  under  fee-for-service  Medicare,  which  requires  determinations  of 
coverage  and  medical  necessity. 
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•  The  CNO  could  choose  to  provide  additional  services  not  traditionally  covered  by 
Medicare,  such  as  prevention  and  health  promotion,  if  these  were  judged  to  be  effective 
for  the  enrolled  population. 

•  More  frequent  screening  (via  the  six-month  reassessment)  could  identify  some  conditions 
at  an  earlier  point  than  in  its  absence. 

Because  the  literature  is  a  poor  guide  to  the  effects  of  these  mechanisms  on  health  outcomes,  few 
clear  hypotheses  emerge.  Two  themes,  however,  stand  out  from  a  review  of  the  literature:  capitation 
payments  are  thought  to  provide  an  incentive  to  reduce  the  cost  of  care,  and  evidence  is  mixed  on 
whether  case  management  and  periodic  assessment  are  likely  to  improve  outcomes. 

1.4.1.        Capitation  and  financial  incentives 

CNO  services  were  financed  through  capitation  payments,  an  arrangement  that  removes  the  link 
between  service  provision  and  payment  and  also  affords  the  CNOs  increased  discretion  in  matching 
services  to  enrollee  needs.  CNOs  have  a  financial  incentive  to  provide  fewer  services  than  they 
would  if  they  were  paid  separately  for  each  service.  In  the  only  study  to  date  comparing  Medicare 
home  health  care  under  HMO  and  fee-for-service  (FFS)  arrangements,  Schlenker,  Shaughnessy  and 
Hittle  (1995)  found  evidence  that  providers  responded  to  these  incentives.  Among  Medicare 
beneficiaries  who  received  home  health  care,  those  who  were  enrolled  in  Medicare  risk  HMOs 
received  fewer  home  health  visits  on  average  than  beneficiaries  who  remained  under  fee-for-service 
Medicare,  even  after  adjustment  for  casemix,  location,  and  demographic  characteristics.  In  a  separate 
article,  Shaughnessy,  Schlenker  and  Hittle  (1994)  found  that  these  same  beneficiaries  experienced 
somewhat  better  outcomes  under  fee-for-service,  leading  them  to  argue  that  "most  HMO  patients  are 
underserved  in  terms  of  the  number  of  home  health  visits." 

It  should  be  noted  that  the  service  package  and  payment  structure  faced  by  the  CNOs  could  produce 
stronger  financial  incentives  to  restrict  services  than  those  faced  by  the  HMOs  studied  by 
Shaughnessy,  Schlenker  and  Hittle.  Most  acute  care  services  covered  by  Medicare  (in  particular 
hospital  and  physician  services)  were  outside  the  CNO  service  package.  Hence  in  contrast  to 
Medicare  risk  HMOs,  at  least  some  portion  of  any  financial  consequences  of  adverse  outcomes 
resulting  from  a  reduction  in  services  (relative  to  FFS)  would  not  be  borne  by  the  CNO.  Consider  for 
example  a  CNO  and  a  Medicare  risk  HMO  each  contemplating  the  provision  of  home  care  costing 
$200  to  a  member.  Suppose  that  both  providers  believe  that  this  care  will  reduce  the  probability  that 
the  member  is  hospitalized  in  the  current  month  from  0.3  to  0.2.  Both  providers  will  incur  a  cost  of 
$200  by  providing  the  care.  The  expected  financial  benefit  from  providing  the  care  is  0.1  times  the 
cost  of  the  hospitalization  for  the  HMO.  The  expected  financial  benefit  to  the  CNO  is  zero.  This 
argument  does  not  imply  that  the  CNO  would  fail  to  provide  the  care  in  question  —  only  that  the 
financial  incentives  to  provide  the  care  are  weaker  for  the  CNO  than  for  the  HMO. 

Although  capitation  does  reduce  the  incentive  to  provide  services,  it  also  permits  greater  flexibility 
for  the  provision  of  services  that  the  CNO  case  manager  considers  most  appropriate,  even  if  the 
services  are  not  covered  by  the  Medicare  fee-for-service  program.  These  may  include  homemaker 
services,  preventive  care,  health  promotion  classes  (e.g.,  smoking  cessation,  cholesterol  and  weight 
control,  exercise  classes,  etc.)  or  telephone  consultations.  Therefore  while  we  may  hypothesize  that 
the  number  of  Medicare-covered  home  health  visits  per  month  or  the  proportion  of  individuals 
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receiving  durable  medical  equipment  (DME)  will  be  lower  among  CNO  enrollees  than  among  the 
control  group,  this  does  not  imply  that  enrollees  necessarily  received  fewer  total  services  or  that  these 
services  are  of  lesser  value  or  effectiveness  than  those  received  by  the  control  group. 

1.4.2.        Case  management  and  periodic  assessment 

Whether  nurse  case  management  can  be  expected  to  markedly  improve  the  health  of  CNO  members 
or  the  cost-effectiveness  of  their  care  is  difficult  to  predict.  The  relevant  literature  provides  little 
guidance  on  the  issue.  The  benefits  claimed  for  case  management  are  typically  rooted  in  the  assertion 
that  health  services  to  a  substantial  portion  of  the  elderly  are  heavily  fragmented.  But  evidence  that 
such  fragmentation  seriously  compromises  care  has  been  difficult  to  find  because  of  the  paucity  of 
studies  directly  comparing  case-managed  and  non-case  managed  elderly  populations.  There  have 
been  several  studies  comparing  alternative  approaches  to  case  management  (Eggert  et  al.,  1991)  or 
evaluating  the  internal  efficiency  of  resource  use  by  case  managers  (Davidson,  Muscovice,  and 
McCaffrey,  1989).  However,  most  studies  that  compare  the  effect  of  case  management  on  a 
treatment  group  against  a  control  group  without  case  management  were  limited  to  psychiatric 
populations  and  have  thus  limited  generalizability  (e.g.,  Jerrell  and  Hu,  1989). 

More  recently,  Burns,  Lamb  and  Wholey  (1996)  found  that  provision  of  nurse  case  management 
services  during  and  after  hospitalization  to  certain  high-risk  members  of  a  senior  risk  plan  resulted  in 
a  significant  reduction  in  subsequent  hospitalizations  and  outpatient  visits.  A  critical  feature  of  the 
case  management  system  studied  by  the  authors  was  targeting  of  individuals  believed  to  be  at  high 
risk.  In  contrast,  Gagnon  et  al.  (1999)  recently  reported  on  a  randomized  trial  of  case  management 
versus  usual  care.  They  found  that  frail  older  people  receiving  nurse  case  management  were  more 
likely  to  use  emergency  health  services  without  a  concomitant  increase  in  health  benefits. 

While  there  is  little  direct  evidence  on  the  subject,  a  consensus  appears  to  have  formed  that  effective 
case  management  requires  successful  targeting.  Eggert  et  al.  (1991)  argued  that  the  success  of  the 
team  model  of  case  management  relied  in  part  on  targeting  a  "high  use/high  cost  group."  And 
Kemper  (1988)  among  others,  argued  that  failure  to  target  services  properly  contributed  to  the 
absence  of  significant  results  in  the  Channeling  demonstration. 

For  the  most  part,  the  individual  CNO  sites  were  free  to  develop  nurse  case  management  and  tailor  it 
to  the  needs  of  the  enrolled  population.  One  element  of  case  management  under  the  CNO,  a  health 
assessment,  conducted  in  person  every  six  months,  was  required  for  all  members.  Periodic 
assessment  of  the  elderly  has  been  examined  in  several  studies  with  conflicting  results.  Tulloch  and 
Moore  (1979)  reported  that  after  two  years,  a  randomly  chosen  group  of  patients  aged  70  and  over 
showed  no  significant  change  in  functional  or  medical  disorders  relative  to  a  control  group. 
Nevertheless,  the  authors  reported  that  "there  was  some  evidence  to  suggest  that  they  were  kept 
independent  for  longer  and  when  admitted  to  hospital,  their  duration  of  stay  was  significantly  shorter 
than  control  group  patients."  Hendriksen,  Lund  and  Stromgard  (1984)  found  stronger  evidence  for 
beneficial  effects  of  screening  in  a  randomized  trial  conducted  among  individuals  aged  75  and  over  in 
a  suburb  of  Copenhagen,  Denmark.  Members  of  the  treatment  group  were  visited  in  their  homes 
every  three  months.  After  three  years,  the  treatment  group  was  found  to  have  experienced  lower 
mortality,  lower  probability  of  hospital  admission,  and  a  strong  suggestion  of  reduced  use  of 
emergency  medical  service.  No  differences  were  found  in  the  number  of  physician  visits  or  home 
nursing  visits.  In  a  similarly  designed  three-year  study,  van  Rossum  et  al.  (1993)  found  no  effect  of 
home  visits  four  times  per  year  on  the  health  of  study  subjects  aged  75-84.  Further  analysis  of  the 
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data,  however,  identified  dramatic  treatment  effects  among  those  who  had  initially  rated  their  health 
as  poor.  The  treatment  group  averaged  20  hospital  days  per  person  over  the  three-year  period  versus 
39  for  the  control  group. 

The  aforementioned  studies,  while  suggestive,  need  not  bear  directly  on  expectations  for  the  current 
CNO  demonstration  since  they  described  interventions  that  were  more  rigid  than  the  current  one.  To 
the  extent  that  CNO  sites  effectively  targeted  and  individualized  their  prevention  and  health 
promotion  activities  to  their  served  populations,  their  outcomes  and  cost-effectiveness  could  turn  out 
to  be  superior  to  those  observed  in  earlier  studies.  Of  particular  interest  is  the  fact  that  each  site  had 
substantial  discretion  about  its  model  of  care  delivery  so  that  differential  effects  across  sites  might  be 
identifiable. 
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2.    Description  of  Phase  II  Sample 


2.1.   Identifying  Eligible  Beneficiaries 

For  members  of  the  treatment,  control,  and  population-based  reference  groups,  demonstration 
eligibility  had  to  be  assessed  every  month  over  the  course  of  the  observation  period.  The  primary 
reason  for  this  was  operational  rather  than  relevant  to  the  evaluation  of  the  demonstration:  since  CMS 
was  paying  CNO  sites  monthly  capitation  and  management  fees  for  enrolled  participants,  it  was 
important  to  ascertain  how  many  beneficiaries  were  actually  enrolled  and  eligible  to  remain  enrolled 
each  month.  Since  CMS  staff  performed  this  screening  only  on  CNO  enrollees,  we  set  out  to  maintain 
comparability  by  excluding  from  our  analysis  all  claims  and  utilization  data  from  months  during 
which  the  beneficiary  would  have  been  deemed  ineligible  had  they  been  enrolled.  However, 
identifying  and  excluding  such  ineligible  months  presented  a  challenge  because  CMS  staff  decisions 
for  enrollees  were  based  on  direct  exchanges  with  site  staff  so  the  underlying  information  was  quite 
accurate.  Since  the  sites  did  not  follow  control  group  participants  or  treatment  group  members  who 
were  not  currently  enrolled,  let  alone  the  population  reference  group,  the  same  information  was  not 
available  for  them. 

To  address  this  problem  when  it  first  arose,  the  initial  evaluation  project  team  constructed  a 
computerized  algorithm  to  screen  the  control  group  in  a  way  that  emulated  as  much  as  possible  CMS 
staffs  eligibility  decisions  for  the  treatment  group.  The  algorithm  had  a  separate  component  that 
attempted  to  identify  months  potentially  ineligible  for  each  of  the  following  reasons: 

•  Medicare  entitlement  based  on  end-stage  renal  disease  only 

•  Ineligibility  for  either  Part  A  or  Part  B 

•  Enrollment  in  a  Medicare  risk  HMO 

•  Residency  in  a  hospice 

•  Residency  in  a  nursing  home  for  more  than  60  days  (30  days  beginning  in  1998) 

•  Residency  outside  the  catchment  area  for  the  CNO 

For  many  of  these  criteria,  however,  the  algorithm  could  produce  at  best  an  imprecise  estimate  of  the 
eligibility  as  determined  by  CMS  staff.  For  example,  it  proved  extremely  difficult  to  emulate 
residency  in  a  nursing  home  on  the  basis  of  Medicare  administrative  data,  since  there  is  no  explicit 
variable  for  nursing  home  residency  available.  Furthermore,  since  Medicare  does  not  pay  for  lengthy 
stays  in  a  nursing  home  but  only  for  post-acute  stays  in  a  skilled  nursing  facility,  there  are  no 
Medicare  claims  for  long-term  nursing  home  care.  The  only  proxy  for  nursing  home  residence  of 
more  than  60  days  that  could  be  constructed  from  Medicare  data  was  based  on  at  least  one  claim  for 
the  beneficiary  that  detailed  a  skilled  nursing  facility  as  place  of  service.  In  contrast,  CMS  staff  were 
able  to  ask  each  beneficiary's  caseworker  directly  and  get  more  accurate  information  on  actual 
nursing  home  stays.  Given  the  differences  between  the  effects  of  the  algorithm  and  of  CMS  staff 
decisions,  the  eligibility  screen  had  to  be  applied  to  the  treatment  group  as  well  as  the  control  group 
for  consistency.  Consequently,  some  treatment  group  months  that  had  been  deemed  eligible  by  CMS 
staff  were  excluded  by  the  algorithm,  but  comparable  months  were  also  excluded  from  the  control 
group.  In  order  to  define  the  best  feasible  eligibility  screen,  we  evaluated  the  performance  of  each  of 
its  components  on  two  criteria,  using  CMS  staff  assessments  as  reference: 
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1 .  False  Positive  Rate:  The  component  should  eliminate  as  few  as  possible  of  the  enrolled 
treatment  group  months  that  CMS  staff  have  classified  as  eligible. 

2.  Sensitivity:  For  a  component  to  be  effective,  it  should  remove  considerably  more  months 
from  the  control  group  than  from  the  enrolled  months  group. 

The  results  of  applying  each  screen  component  to  a  subsample6  of  the  data  are  summarized  in  Table 
2.1.1.  The  table  contrasts  the  effect  of  each  screen  component  on  control  group  months  with  its  effect 
on  actually  enrolled  months  because  these  were  the  only  months  during  which  CMS  staff  and  the  sites 
actually  made  a  determination  of  eligibility. 

The  result  for  both  Part  A  and  Part  B  ineligibility  was  unambiguous:  Neither  component  of  the  screen 
eliminated  any  enrolled  months,  i.e.  they  had  a  zero  false  positive  rate,  but  a  substantial  number  of 
control  group  months  were  excluded,  i.e.  their  sensitivity  was  sufficient.  Similarly,  the  Medicare 
HMO  component  eliminated  almost  no  enrolled  months  but  a  high  number  of  control  group  months. 
This  result  was  not  surprising  since  those  three  criteria  are  unambiguously  defined  in  the  Medicare 
Enrollment  Database.  We  conclude  that  these  three  components  should  clearly  be  applied.  It  should 
be  noted  that  since  a  beneficiary  would  not  generate  any  Medicare  claims  in  these  months,  failure  to 
exclude  such  months  would  have  led  us  to  code  expenditures  incorrectly  as  zero  rather  than  as 
missing  or  censored,7  resulting  in  too  low  an  estimate  of  average  monthly  outlays.  We  also  decided 
to  retain  the  hospice  residency  component  since  it  excluded  more  control  group  months  than  eligible 
months  both  in  absolute  terms  and  as  a  percentage. 

The  "out-of-area"  residency  screen  component  did  not  perform  as  well  as  hoped,  probably  because 
we  used  the  official  address  of  the  beneficiary.  This  might  not  correspond  to  the  actual  place  of 
residency,  in  particular  for  beneficiaries  who  migrate  seasonally  ("snowbirds"  and  "sunbirds"). 
However,  we  had  to  retain  this  criterion,  because  the  Medicare  Part  B  claims  data  for  the  population 
reference  group  had  been  retrieved  through  the  Medicare  Part  B  State  Files  rather  than  the  National 
Claims  History  100%  File.  Thus,  without  the  residency  exclusion,  we  would  have  missing  data  for 
Part  B  claims  for  beneficiaries  who  moved  into  a  state  without  a  CNO  site. 


6  This  analysis  was  performed  under  a  previous  contract  that  only  examined  data  from  Waves  1  and  2  of  the 
demonstration. 

7  Obviously,  the  Medicare  program  would  still  incur  costs  for  a  beneficiary  enrolled  in  a  Medicare  HMO  that 
should  be  included  in  a  comparison  of  expenditures.  However,  as  the  present  evaluation  had  no  access  to  data 
on  payments  to  Medicare  HMOs,  we  were  operationally  unable  to  account  for  these  costs. 
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Table  2.1.1 

Examination  of  the  effect  of  eligibility  screening  algorithms  on  exclusion  of  analyzed  months 


Enrolled  Months  Excluded  Control  Group  Months  Excluded 


Eligibility  Screen 

Number 

Percent 

Number 

Percent 

Part  A  Ineligibility 

0 

0.00% 

45 

0.04% 

Part  B  Ineligibility 

0 

0.00% 

164 

0.14% 

ESRD  Eligibility 

182 

0.09% 

138 

0.12% 

Medicare  HMO 

52 

0.03% 

4013 

3.47% 

Hospice  Residency 

221 

0.11% 

347 

0.30% 

Nursing  Home 

1272 

0.65% 

991 

0.86% 

Residency 

Residency  out  of  Area 

6494 

3.34% 

6478 

5.60% 

The  denominators  are  total  numbers  of  non-missing  months  (Enrolled  Months  n= 

194,496,  Control  Group  Months  n= 

115,577).  ESRD 

denotes  end  stage  renal  disease,  HMO  health  maintenance  organization. 

Sources:  CNO  Enrollment  File,  Medicare  Enrollment  Database,  January  1 994-December  1996 


The  remaining  two  components  did  not  seem  useful,  since  they  eliminated  similar  shares  of  enrolled 
and  control  group  months.  Thus,  they  appeared  not  to  emulate  CMS  staff  decisions  but  to  impose  a 
different  decision  rule.  This  was  not  a  surprising  outcome  for  the  nursing  home  residency  screen.  As 
outlined  above,  our  ability  to  operationalize  this  criterion  on  the  basis  of  administrative  data  was  quite 
limited.  By  contrast,  the  fact  that  the  ESRD  screen  eliminated  a  similar  number  of  enrolled  and 
control  group  months  was  more  of  a  surprise  because  both  our  algorithm  and  CMS  staff  decisions 
were  based  on  information  in  the  Medicare  EDB.  However,  since  our  algorithms  were  applied  months 
or  years  after  CMS  staff  decisions,  changes  in  the  EDB  over  time  probably  explain  this  observation. 
Since  we  were  unable  to  reproduce  CMS  staff  decisions  with  respect  to  ESRD,  this  component  of  the 
eligibility  screen  could  not  be  retained. 

These  results  suggested  that  the  two  components  to  add  to  the  eligibility  screen  beyond  those  which 
remove  months  without  Medicare  claims  (Part  A  and  Part  B  eligibility  and  enrollment  in  a  Medicare 
HMO)  were  hospice  residency  and  out-of-area  residency.  Given  that  the  available  data  did  not  permit 
us  to  emulate  CMS  staff  assessments  on  ESRD  eligibility  and  residency  in  a  nursing  home,  those 
components  were  removed  from  the  eligibility  screen. 

2.2.  Enrollment 

As  described  in  Section  1 .2.2,  CNO  applicants  were  randomized  to  treatment  or  control  status  in  a  2: 1 
ratio  during  Wave  1  and  Wave  3  of  the  demonstration.  This  arrangement  allowed  sites  to  build  up 
enrollee  numbers  more  quickly  in  order  to  be  able  to  provide  their  full  range  of  services.  In  addition, 
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all  applicants  were  assigned  to  the  treatment  group  during  Wave  2  and  Wave  4.  Thus,  three-quarters 
of  the  14,850  demonstration  participants  were  assigned  to  CNO  treatment. 


Table  2.2.1 


Number  and  percentage  of  treatment  and  control  group  participants  by  site 


Site 

Treatment  Group 

Control  Group 

Number 

Percent 

Number 

Percent 

Carle  Clinic 

3,191 

76% 

1,000 

24% 

Carondelet 

3,691 

74% 

1,322 

26% 

LAH 

2,196 

72% 

868 

28% 

VNS 

1,988 

77% 

594 

23% 

Total 

11,066 

75% 

3,784 

25% 

Sources:    CNO  Enrollment  File,  January  1994-December  1999 


The  number  of  participants  by  site  and  treatment/control  status  is  summarized  in  Table  2.2.1.  This 
group  will  be  the  subject  of  our  first  set  of  comparisons  of  average  spending  and  utilization,  presented 
in  Chapter  3,  and  will  be  referred  to  as  the  participant  group. 


Table  2.2.2 


Number  and  percentage  of  treatment  and  control  group  participants  by  wave 


Wave 

Treatment  Group 

Control  Group 

Number 

Percent 

Number  Percent 

1 

7,138 

67% 

3,508  33% 

2 

1,016 

100% 

4a  0% 

3 

885 

76% 

272  24% 

4 

2,027 

100% 

0  0% 

Total 

11,066 

75% 

3,784  25% 

3    The  four  beneficiaries  assigned  to  the  control  group  during  Wave  2  may  have  been  Qualified  Household  Members  who  were  unaware 

that  their  spouse  had  previously  been  assigned  to  control  status. 


Sources:    CNO  Enrollment  File,  January  1994-December  1999 


The  intermittent  nature  of  randomization  can  be  seen  in  Table  2.2.2,  which  shows  enrollment  in 
treatment  and  control  groups  by  wave.  As  indicated  by  the  table,  enrollees  from  Waves  2  and  4  had 
no  control  group,  making  a  comparison  of  means  between  treatment  and  control  potentially 
misleading.  To  see  why  this  might  be  the  case,  consider  that  new  CNO  enrollees  were  generally 
younger  than  those  who  had  enrolled  earlier.  Consequently,  if  a  mean  calculated  from  the  entire 
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treatment  group  were  compared  to  a  mean  calculated  from  the  entire  control  group,  the  control  group 
would  be  older  on  average  and  therefore  would  be  expected  to  have  higher  per  person  expenditures. 

When  initial  assignment  to  the  treatment  and  control  groups  is  plotted  against  time  elapsed  since  the 
start  of  the  demonstration  in  January  1994,  the  assignment  patterns  of  the  four  waves  become  easily 
visible.  Figures  2.1  through  2.4  illustrate  assignment  over  time  for  each  of  the  four  sites.  One  can  see 
that  the  treatment  group  grows  much  faster  than  the  control  group,  and  that  the  control  group  has  two 
long  periods  without  any  new  assignments  (Waves  2  and  4),  interrupted  by  a  short  period  of 
additional  assignment  (Wave  3).  Thus,  the  treatment  group  is  almost  two-and-one-half  times  as  large 
as  the  control  group  by  the  end  of  the  observation  period  (compare  the  top  and  bottom  lines). 


Figure  2.1 

Treatment  and  control  groups  by  month:  Carle  Clinic 
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Control  group  contains  both  enrolled  and  unenrolled  beneficiaries. 
Sources:  CNO  Enrollment  File 


Also  shown  is  actual  enrollment  from  the  treatment  group  and  from  treatment  and  control  groups 
combined.  It  is  clear  from  these  figures  that  the  sites  maintained  relatively  steady  enrollment  through 
the  end  of  1999,  replenishing  the  treatment  group  as  members  disenrolled.  Since  the  control  group 
was  not  replenished  in  this  way  during  Waves  2  and  4,  the  resulting  compositional  change  in  the 
treatment  group  is  a  potential  source  of  bias  when  unadjusted  means  are  compared.  Finally,  the 
contamination  of  the  control  group  is  evident  in  the  number  of  controls  who  enrolled  (difference 
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between  middle  two  lines).  By  January  of  1999,  more  than  half  of  the  control  group  at  Carle  Clinic 
had  been  enrolled,  substantially  compromising  the  value  of  randomization. 

V 

Figure  2.2 

Treatment  and  control  groups  by  month:  Carondelet 
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Control  group  contains  both  enrolled  and  unenrolled  beneficiaries. 
Sources:  CNO  Enrollment  File 


The  cyclical  pattern  of  actual  enrollment  at  Carondelet  reflects  seasonal  migration  by  beneficiaries 
("snowbirds"  and  "sunbirds"),  a  phenomenon  that  was  particularly  pronounced  at  the  Arizona  site 
where  enrollment  declined  every  summer. 
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Figure  2.3 

Treatment  and  control  groups  by  month:  LAH 
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Control  group  contains  both  enrolled  and  unenrolled  beneficiaries. 

Sources:  CNO  Enrollment  File 
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Figure  2.4 

Treatment  and  control  groups  by  month:  VNS 
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2.3.  Comparison  of  Reference  Group  to  Participants 

As  discussed  in  Section  1 .2.4,  the  randomized  design  of  the  Phase  I  Evaluation  permitted  a  simple 
comparison  of  means  as  a  measure  of  the  treatment  effect,  whereas  nonrandom  selection  in  the  Phase 
II  Evaluation  necessitated  adjustment  for  differences  between  treatment  and  reference  groups.  Since 
the  adjustment  process  was  based  on  complex  statistical  models  that  are  explained  in  detail  in  Section 
3.1.3,  a  much  larger  sample  size  than  the  number  of  demonstration  participants  was  required. 
Therefore,  a  population-based  reference  group  was  created  consisting  of  all  eligible  Medicare 
beneficiaries  who  lived  in  a  CNO  catchment  area.  The  characteristics  of  this  reference  group  are 
described  in  the  following  tables  and  compared  to  the  demonstration  participants  for  each  site 
separately. 

In  1994,  the  first  year  of  the  CNO  demonstration,  the  treatment  and  control  groups  were  similar  with 
respect  to  age,  sex  and  race  (Table  2.3.1).  This  is  as  expected,  since  a  substantial  number  of  them  had 
been  assigned  randomly.  Demonstration  participants  appeared  to  be  older  on  average  than  the 
population  due  to  the  lack  of  non-elderly  disabled  Medicare  beneficiaries  in  the  demonstration. 


Table  2.3.1 


Demographic  characteristics  of  the  treatment,  control  and  population  reference  groups, 
first  year  of  CNO  demonstration  (1994) 


Site 

Status 

N 

Mean  Age 

Elderly 

Mean  Age 

Female 

Non-wh 

% 

of  Elderly 

% 

% 

Carle 

Treatment 

3,191 

72.21 

88% 

73.84 

62% 

3% 

Carle 

Control 

1,000 

72.23 

94% 

73.01 

60% 

2% 

Carle 

Population 

86,407 

70.81 

82% 

75.08 

58% 

6% 

Carondelet 

Treatment 

3,691 

72.60 

90% 

74.01 

62% 

3% 

Carondelet 

Control 

1,322 

73.55 

93% 

74.50 

60% 

3% 

Carondelet 

Population 

124,594 

70.35 

82% 

74.29 

55% 

10% 

LAH 

Treatment 

2,196 

74.62 

92% 

75.74 

68% 

1% 

LAH 

Control 

868 

74.96 

97% 

75.33 

66% 

0% 

LAH 

Population 

164,029 

70.92 

83% 

'  75.24 

59% 

5% 

VNS 

Treatment 

1,988 

76.35 

95% 

77.18 

77% 

6% 

VNS 

Control 

594 

76.93 

98% 

77.21 

78% 

7% 

VNS 

Population 

643,856 

71.50 

84% 

75.20 

60% 

29% 

Elderly  refers  to  those  over  the  age  of  65. 

Sources:    CNO  Enrollment  File,  Medicare  Enrollment  Database 
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When  the  comparison  is  restricted  to  Medicare  beneficiaries  over  the  age  of  65  (Mean  Age  of  Elderly 
column),  participants  and  the  population  were  similar  with  respect  to  age.  In  addition  to  the  disabled, 
Table  2.3.1  indicates  that  men  and  nonwhites  were  underrepresented  in  all  four  demonstration  sites 
relative  to  the  population. 

To  compare  utilization  and  spending,  we  tabulated  Medicare  claims  from  1993,  the  year  before  the 
inception  of  the  demonstration,  to  eliminate  any  possible  demonstration  effect.  With  the  exception  of 
treatment  group  members  in  LAH,  participants  had  higher  spending  on  average  than  the  Medicare 
population,  both  for  CNO-provided  and  for  non-CNO  services  (Table  2.3.2).8  Participants  also  had 
more  physician  office  visits,  emergency  room  days  and  hospital  admissions.  This  more  intense  use  of 
medical  resources  is  to  some  degree  surprising,  since  previous  analyses  had  shown  that  participants 
had  lower  age-adjusted  mortality  rates,  indicating  better  health  status  (Abt  Associates  Inc.  1998). 


Table  2.3.2 

* 

Spending  and  utilization  characteristics  of  the  treatment,  control  and  population  reference 
groups,  first  year  prior  to  CNO  demonstration  (1993) 


Expenditures:  Average  Medicare       Utilization:  Average 
Spending  Number  of 


Site 

Status 

N 

Total 

In- 

Out-of- 

Admis- 

ER 

MD 

Bundle 

Bundle 

sions 

Days 

Visits 

Carle 

Treatment 

3,191 

$2,033 

$230 

$1,803 

0.18 

0.20 

5.32 

Carle 

Control 

1,000 

$2,316 

$208 

$2,108 

0.21 

0.15 

5.11 

Carle 

Population 

86,407 

$1,865 

$182 

$1,683 

0.20 

0.18 

3.61 

Carondelet 

Treatment 

3,691 

$3,070 

$253 

$2,817 

0.23 

0.21 

7.41 

Carondelet 

Control 

1,322 

$3,205 

$274 

$2,931 

0.22 

0.21 

7.69 

Carondelet 

Population 

124,594 

$2,224 

$252 

$1,973 

0.15 

0.17 

3.97 

LAH 

Treatment 

2,196 

$1,860 

$105 

$1,754 

0.19 

0.15 

4.97 

LAH 

Control 

868 

$2,372 

$160 

$2,212 

0.22 

0.20 

4.80 

LAH 

Population 

164,029 

$2,147 

$192 

$1,954 

0.17 

0.20 

3.07 

VNS 

Treatment 

1,988 

$4,151 

$500 

$3,651 

0.23 

0.13 

8.44 

VNS 

Control 

594 

$4,362 

$561 

$3,800 

0.23 

0.16 

8.64 

VNS 

Population 

643,856 

$3,586 

$337 

$3,249 

0.20 

0.13 

4.79 

In-bundle  and  out-of-bundle  spending  are  as  defined  in  Section  1 . 1 .3;  admissions  denotes  inpatient  hospital  admissions;  ER  days 
refers  to  the  number  of  days  during  which  there  was  an  emergency  room  claim;  and  MD  visits  refers  to  the  number  of  physician 
visits. 

Sources:  CNO  Enrollment  File,  Medicare  Enrollment  Database,  National  Claims  History  File 


CNO-provided,  or  "in-bundle"  services  under  the  demonstration  are  described  in  Section  1.1.3. 
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One  reason  why  average  expenditures  would  be  higher  for  CNO  demonstration  participants  is  the  fact 
that  participants  were  more  likely  than  the  population  to  have  nonzero  Medicare  claims  in  a  given 
year.  As  illustrated  in  Table  2.3.3,  only  about  one  percent  of  the  treatment  group  had  zero  claims.  By 
contrast,  between  14  and  20  percent  of  the  population  had  zero  Medicare  utilization.  These  figures 
show  that  beneficiaries  with  minimal  utilization  were  unlikely  to  apply  to  the  demonstration,  leading 
to  adverse  selection.  This  phenomenon  was  mitigated  by  the  tendency  of  beneficiaries  with  high 
utilization  to  drop  out  of  the  demonstration  (or  not  apply  in  the  first  place),  as  we  will  show  in 
Chapter  3  (see  Table  3.3.5). 


Table  2.3.3 


Percentage  of  Medicare  Beneficiaries  without  any  Medicare  Claims  in  Each  Year  of  the 
Demonstration 


Carle 

Carondelet 

LAH 

VNS 

Year 

Status 

1994 

Treat 

2% 

2% 

5% 

2% 

Pop 

17% 

19% 

21% 

18% 

1995 

Treat 

1% 

2% 

1% 

2% 

Pop 

15% 

18% 

21% 

18% 

1996 

Treat 

1% 

1% 

1% 

1% 

Pop 

15% 

19% 

20% 

19% 

1997 

Treat 

1% 

1% 

1% 

1% 

Pop 

14% 

19% 

20% 

19% 

1998 

Treat 

1% 

1% 

1% 

1% 

Pop 

13% 

17% 

20% 

19% 

1999 

Treat 

1% 

1% 

1% 

0% 

Pop 

12% 

17% 

18% 

19% 

Average 

Treat 

1% 

1% 

2% 

1% 

Pop 

14% 

18% 

20% 

19% 

Treat  denotes  treatment  group;  Pop  denotes  population  reference  group.  Figures  for  the  control  group  were  similar  to 
those  for  the  treatment  group. 


Sources:    Abt  Associates  Inc.  CNO  Evaluation  Analytic  Files 


Any  two  groups  not  generated  by  random  assignment  are  likely  to  be  different,  so  the  fact  that  the 
population  reference  group  systematically  differed  from  the  treatment  group  should  come  as  no 
surprise.  It  is  precisely  these  differences  that  can  lead  to  selection  bias,  requiring  the  application  of 
statistical  techniques,  discussed  in  detail  in  Appendix  A,  to  correct. 
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3.    Analysis  of  Randomized  Beneficiaries 
(Treatment  to  Control  Contrasts) 

Between  January  1  1994  and  September  30  1995  and  again  between  January  1  and  October  2  1997, 
applicants  to  the  four  CNO  programs  were  randomized  to  treatment  or  control  status.  Though  difficult 
to  implement  in  many  demonstration  environments,  randomization  is  a  powerful  tool  for  avoiding 
bias  when  estimating  its  effects.  Randomization  allows  us  to  assert  that  in  the  absence  of 
demonstration  effects,  the  expected  value  of  any  specified  outcome  (say,  Medicare  outlays  per 
person-month)  for  the  treatment  group  is  equal  to  the  expected  value  of  the  outcome  for  the  control 
group.  Therefore  any  systematic  difference  in  outcomes  between  the  treatment  and  control  groups  can 
be  ascribed  to  the  demonstration. 

This  chapter  uses  the  experience  of  randomized  beneficiaries  to  estimate  the  effect  of  the  CNO  on 
total  Medicare  outlays  per  month  and  on  utilization  of  selected  categories  of  service.  Because  of  the 
advantages  of  randomization,  the  results  in  this  chapter  probably  constitute  the  most  reliable  and 
accurate  evidence  in  this  report.  The  analysis  suggests,  as  we  shall  see,  that  the  CNO  tended  to 
increase  Medicare  expenditures  by  5  to  15  percent  and  that  utilization  of  services  not  covered  by  the 
CNO  was  nearly  identical  for  treatment  and  control  groups.  Further  analyses  of  CNO  effects, 
presented  in  Chapter  4,  dispense  with  randomization  and  compare  CNO  enrollees  with  eligible  non- 
enrollees  from  the  same  locality.  Those  analyses  use  Hierarchical  Coexisting  Conditions  (HCCs)  and 
so-called  "difference-in-differences  models"  to  adjust  beneficiary  outcomes  for  variations  in 
underlying  health  and  risk.  The  conclusions  of  that  analysis,  while  qualitatively  similar  to  those  of 
this  chapter,  differ  in  magnitude  and  in  the  distribution  of  effects  across  CNO-covered  and  non-CNO- 
covered  services.  Methodological  details  relating  to  both  this  chapter  and  Chapter  4  can  be  found  in 
Appendix  A. 

3.1.  Introduction 

The  net  saving  (positive  or  negative)  of  the  CNO  intervention  for  the  Medicare  program  is  estimated 
here  by  contrasting  total  Medicare  expenditures  for  the  treatment  and  control  groups  from  the  time  of 
randomization  through  December  1999.9  If  the  CNO  capitation  rate  and  case  management  fee  were 
set  to  be  no  greater  than  the  expected  value  of  monthly  Medicare  outlays  for  CNO-covered  services  in 
the  fee-for-service  sector  and  if  enrollment  in  the  CNO  does  not  lead  to  an  increase  in  use  of  non- 
CNO  (in  particular  hospital  and  physician)  services,  then  total  Medicare  expenditures  per-person  per- 
month  for  the  treatment  group  should  be  less  than  or  equal  to  that  of  the  control  group. 

As  noted  earlier,  previous  analyses  have  found  that  total  Medicare  expenditures  for  beneficiaries 
assigned  to  the  treatment  group  exceeded  expenditures  for  those  assigned  to  the  control  group.  This 
result  implies  that  the  CNOs  led  to  increased  cost  for  the  Medicare  program.  The  results  presented 
here  extend  the  earlier  analyses  by  using  data  for  up  to  72  months  after  randomization  and  by 
measuring  the  effects  of  the  CNO  not  only  on  overall  Medicare  expenditures,  but  also  on  utilization 
and  expenditures  for  inpatient  hospital  stays,  emergency  room  use,  and  physician  office  visits. 


The  Final  Report  will  extend  the  analysis  through  December  2000. 
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We  acknowledge  at  the  outset  a  difficulty  with  the  empirical  strategy  that  did  not  affect  earlier 
analyses.  In  October  1997,  members  of  the  control  group  were  allowed  to  enroll  in  the  CNO  if  they 
wished  to  do  so.  Because  some  members  of  the  control  group  received  CNO  services  after  October 
1997,  the  control  group  is  considered  contaminated  after  this  point.  Although  the  treatment-control 
differences  are  therefore  biased  estimates  of  the  CNO  effect,  the  effect  of  this  bias  is  shown  in  the 
following  section  to  understate  the  true  CNO  effect.  That  is,  the  measured  effect  is  biased  toward 
zero. 

3.2.  Analytic  approach 

Medicare  claims  for  every  randomized  beneficiary  were  assembled  from  the  month  of  randomization 
until  December  1999  or  the  month  of  death,  whichever  came  earlier.  Hence  a  maximum  of  72  months 
of  expenditure  data  for  each  randomized  person  were  available  for  analysis.  All  person-months  were 
deleted  from  the  analysis  in  which  a  beneficiary  a)  was  not  enrolled  in  Part  A  and  B  of  Medicare,  b) 
was  enrolled  in  a  Medicare  HMO  (cost  or  risk)  or  Health  Care  Prepayment  Plan  (HCPP),  c)  was 
resident  in  a  hospice,  or  d)  resided  in  a  state  other  than  Arizona,  Illinois,  Minnesota,  and  New  York. 
Expenditures  were  classified  as  CNO-covered  or  non-CNO-covered  services,  as  shown  in  Table 
1.3.1.  The  treatment  and  control  groups  were  compared  in  terms  of  a)  total  Medicare  expenditures,  b) 
hospital  utilization  and  expenditures,  c)  emergency  room  utilization  and  expenditures,  and  d) 
physician  office  visit  utilization  and  expenditures. 

In  all  comparisons  between  the  treatment  and  control  groups,  two  separate  definitions  of  the  treatment 
group  were  used.  The  first  defined  the  treatment  group  as  all  individuals  randomized  to  the  treatment 
group,  even  if  they  later  disenrolled  from  the  CNO.  In  so  doing,  it  follows  the  principle  of  "intent  to 
treat"  under  which  study  subjects  are  analyzed  according  to  their  initial  treatment/control  assignment 
regardless  of  compliance  or  noncompliance  with  the  experimental  intervention.  The  second  defined 
the  treatment  group  to  consist  of  person-months  during  which  beneficiaries  were  actually  enrolled  in 
the  CNO,  regardless  of  the  randomization  status  of  the  beneficiary.  Under  this  second  definition, 
individuals  were  retained  in  the  treatment  group  only  for  those  months  during  which  they  were 
actually  enrolled  in  the  CNO.  Cumulative  expenditures  per  person  per  month  (PPPM)  were  computed 
for  both  groups  by  month  of  enrollment.  All  dollar  amounts  are  expressed  in  1999  dollars  using  the 
Consumer  Price  Index  for  discounting. 

Under  intent-to-treat,  the  difference  in  mean  monthly  expenditures  for  the  treatment  and  control 
groups  does  not  estimate  the  expected  saving  to  Medicare  of  a  month  spent  in  the  CNO  but  rather  the 
expected  saving  (positive  or  negative)  associated  with  the  CNO  averaged  over  months  in  which  those 
assigned  to  the  treatment  group  could  have  been  in  the  CNO.  This  measure  of  CNO  effect  is 
appropriate  for  evaluation  of  a  demonstration  intervention  that  might  be  implemented  into  the 
Medicare  program  as  a  voluntary  option  for  the  beneficiary.  Were  a  CNO  program  to  become  part  of 
the  Medicare  benefit,  it  is  highly  unlikely  that  beneficiaries  would  be  compelled  to  remain  in  the 
CNO  once  they  had  joined.  Rather,  beneficiaries  would  be  permitted  to  leave  the  program  and  some 
would  do  so,  just  as  they  did  in  the  demonstration.  The  treatment/control  difference  thus  answers  the 
question,  "For  beneficiaries  likely  to  enroll  in  a  CNO  program,  what  is  the  average  monthly  saving  to 
Medicare  of  giving  them  the  option  to  do  so?" 

Comparisons  that  do  not  follow  the  intent-to-treat  principle  -  in  particular,  comparisons  between 
CNO  enrollees  and  members  of  the  control  group  -  provide  minimal  information  to  the  evaluation. 
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Because  such  comparisons  are  contaminated  by  self-selection,  they  have  no  clear  interpretation  and 
do  not  estimate  any  CNO  effect.  If  beneficiaries  are  more  likely  to  drop  out  of  the  CNO  when  then- 
health  deteriorates  than  at  other  times,  then  the  difference  in  expenditures  between  enrollees  and 
controls  will  overstate  the  cost-saving  effects  of  the  CNO.  If  they  are  less  likely  to  drop  out  when 
their  health  deteriorates,  the  difference  will  understate  these  effects.  Despite  our  belief  that  little  is  to 
be  learned  from  comparisons  between  enrollees  and  controls,  they  are  presented  here,  together  with 
the  treatment/control  comparisons  because  the  sites  have  repeatedly  expressed  interest  in  them  and 
policy  makers  may  wish  to  consider  them. 

Note  that  unlike  earlier  reports,  the  tables  shown  here  do  not  display  means  for  so-called  "trimmed 
expenditures."  Trimming  was  a  means  of  adjusting  for  use  of  out-of-plan  services  by  beneficiaries 
enrolled  in  the  CNO.10  Because,  unlike  in  earlier  reports,  our  expenditure  figures  include  payment 
adjustments  made  by  CMS  staff  in  response  to  out-of-plan  use  (see  Section  1 .3.1),  we  chose  not  to 
calculate  trimmed  expenditures  for  this  report.  In  effect,  calculating  trimmed  expenditures  when 
payment  adjustments  have  already  been  included  would  have  adjusted  CNO  expenditures  twice  for 
the  same  out-of-plan  use. 

Two  objections  have  been  raised  to  the  comparison  of  expenditures  and  utilization  per-person  per- 
month  as  is  done  here.  First,  there  is  a  tendency  for  these  comparisons  to  be  weighted  heavily  by  the 
early  experience  of  program  participants.  Because  CNO  effects  are  likely  to  appear  gradually  over 
time  among  enrollees  and  because  the  number  of  month  1  and  month  2  observations  will  invariably 
exceed  the  number  of,  say,  month  35  and  month  36  observations,  the  estimation  of  CNO  treatment 
group  expenditures  may  understate  true  CNO  effects.  Second,  treatment/control  differences  may  be 
depressed  by  the  inclusion  of  early  years  of  CNO  operation  when  program  staff  had  not  yet  learned 
how  best  to  provide  services  to  a  diverse  clientele.  It  is  reasonable  to  expect  that  a  number  of  lessons 
would  be  learned  during  the  initial  phase  of  operation  that  would  result  in  better  identification  of 
beneficiary  needs  and  improved  service  delivery  in  later  years. 

To  address  these  issues,  expenditures  per-person  per-month  were  recomputed  after  deleting  all  data 
for  the  first  six  months  after  randomization  for  all  beneficiaries  and  after  deleting  data  for  1994,  the 
first  year  of  CNO  operation,  if  "startup  effects"  on  either  beneficiaries  or  the  CNOs  themselves  were 
of  substantial  magnitude,  then  treatment/control  differences  computed  on  this  pared-down  sample 
may  be  a  more  accurate  estimate  of  long-term  CNO  effects. 

Finally,  we  investigate  whether  beneficiaries  assigned  to  the  treatment  group  who  subsequently 
dropped  out  of  the  CNO  are  similar  to  those  who  did  not.  If  beneficiaries  tend  to  leave  the  CNO  when 
their  health  deteriorates  or  when  their  service  utilization  is  otherwise  higher  than  average,  then  we 
expect  to  find  that  mean  monthly  expenditures  over  the  six  months  prior  to  the  time  at  which  they 
leave  the  CNO  would  be  higher  than  mean  monthly  expenditures  for  beneficiaries  who  remain  in  the 
CNO  from  the  time  of  randomization  through  the  end  of  the  data  collection  period. 


To  approximate  expenditures  as  they  would  appear  if  a  lock-out  mechanism  that  prevented  Medicare 
payment  for  out-of-plan  services  were  in  place,  all  claims  for  CNOcovered  services  provided  by  fee-for- 
service  providers  to  individuals  currently  enrolled  in  the  CNO  were  removed  to  construct  the  measure  of 
trimmed  expenditures. 
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Statistical  tests 

Tests  for  statistical  significance  of  differences  in  Medicare  expenditures  per  person-month  are  not 
easy  to  conduct  because  monthly  expenditures  for  a  given  beneficiary  cannot  be  assumed  to  be 
independent  from  month  to  month.  Ignoring  this  month-to-month  dependence  results  in  standard 
errors  that  are  biased  downward,  resulting  in  mistaken  findings  of  statistical  significance  where  none 
are  warranted. 

The  statistical  significance  of  treatment/control  differences  in  total  expenditures  (represented  by  the 
first  two  lines  of  Tables  3.3.1  and  3.3.2)  were  assessed  by  least-squares  regression.  Monthly 
expenditures  were  assumed  to  be  independent  across  beneficiaries.  At  the  beneficiary  level,  expen- 
ditures in  separate  months  were  assumed  to  follow  a  first-order  autoregressive  process.  This  implies 
that  expenditures  in  months  that  are  close  together  are  more  highly  correlated  than  expenditures  in 
months  that  are  far  apart."  Statistical  significance  is  represented  by  asterisks  in  the  top  lines  of  Tables 
3.3.1  and  3.3.2. 

Threats  to  the  analysis 

The  decision  to  allow  members  of  the  control  group  to  enroll  in  the  CNO  after  October  1997  does 
present  a  difficulty  for  the  intent-to-treat  analysis.  Between  October  1997  and  December  1999,  some 
695  beneficiaries  initially  assigned  to  the  control  group  (18  percent  of  all  controls)  enrolled  in  the 
CNO.  These  beneficiaries  remain  in  the  control  group  (as  they  must  under  the  intent-to-treat 
principle),  but  are  considered  to  be  "contaminated  controls"  because  they  have  received  the  CNO 
services  and  so  are  likely  to  exhibit  outcomes  that  are  different  than  those  that  would  have  obtained 
had  they  not  received  these  services.  As  a  result,  treatment/control  differences  that  include  these  data 
will,  in  general,  be  biased.  As  is  shown  in  Appendix  B,  however,  the  resulting  biased  estimate 
understates  the  true  cost  savings,  positive  or  negative,  associated  with  the  CNO. 

3.3.  Results 

Over  the  first  72  months  of  operation  of  the  demonstration,  total  monthly  Medicare  expenditures  per 
person  were  higher  for  the  treatment  group  in  all  of  the  four  sites  as  can  readily  be  seen  in  the  upper 
panel  of  Table  3.3.1.  The  relative  difference  in  total  expenditures  per  month  between  the  treatment 
and  control  groups  varied  from  five  percent  at  Carondelet,  to  over  14  percent  at  the  Illinois  and 
Minnesota  sites.  Although  expenditures  for  non-CNO  services  were  comparable  for  treatment  and 
control  groups  at  three  of  the  sites  and  $20  per-person  per-month  lower  for  treatments  than  controls  at 
the  Carondelet  site,  mean  expenditures  for  CNO-covered  services  were  greater  for  treatments  than  for 
controls  at  every  site  by  amounts  ranging  from  $44  to  $55  per-person  per-month.  These  amounts 
represented  relative  differences  of  between  54  and  180  percent.  The  differences  in  mean  expenditures 
for  CNO  enrollees  vs.  controls,  shown  in  the  third  and  fourth  lines  of  each  panel,  are  slightly  smaller 
than  for  treatments  vs.  controls.  At  Carondelet,  mean  expenditures  are  virtually  the  same,  since  the 
$39  difference  in  monthly  expenditures  for  non-covered  services  nearly  offsets  the  additional  $45  per 
month  spending  for  treatments  on  CNO-covered  services. 


Formally,  the  assumption  that  expenditure  follows  an  autoregressive  process  implies  that  the  covariance  of 
expenditure  between  month  t  and  month  t  ±  j  is  given  by  pV  where  0<p<l  and  a2  is  the  variance  of 
monthly  expenditure. 
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Table  3.3.2  shows  expenditures  per-person  per-month  after  deleting  the  first  6  months  of  data  for  each 
beneficiary  and  deleting  all  data  for  the  year  1994.  As  noted  above,  these  deletions  eliminate  early 
person-months  after  randomization  before  the  CNO  intervention  may  have  had  sufficient  time  to 
affect  a  beneficiary's  Medicare  utilization  and  also  eliminates  data  from  the  first  year  of  CNO 
operation  when  the  CNOs  themselves  might  still  have  been  learning  how  best  to  target  and  deliver 
services  to  their  members.  As  the  table  makes  clear,  however,  the  deletions  had  almost  no  effect  on 
mean  Medicare  expenditures  per-person  per  month.  Mean  expenditures  for  the  treatment  group 
exceeded  that  for  the  control  group  in  every  site,  by  amounts  ranging  from  seven  to  1 5  percent  - 
nearly  identical  to  the  relative  difference  between  treatments  and  controls  shown  in  Table  3.3.1. 

In  most  cases,  the  discrepancy  in  total  Medicare  expenditures  between  treatment  and  control  groups 
was  substantially  smaller  when  the  treatment  group  was  defined  as  beneficiaries  currently  enrolled  in 
the  CNO  than  it  was  when  the  treatment  group  was  defined  as  those  randomly  assigned  to  the 
treatment  group.  As  we  shall  see,  this  is  probably  the  result  of  a  tendency  for  beneficiaries  to  drop  out 
of  the  CNO  in  periods  when  their  Medicare  expenditures  were  especially  high. 

Medicare  expenditures  per-person  per-month  for  hospital  inpatient  care,  emergency  room  care,  and 
physician  office  visits  are  shown  in  Table  3.3.3.  Mean  monthly  expenditures  for  inpatient  care  were 
similar  for  the  treatment  and  control  groups,  ranging  from  four  percent  less  for  the  treatment  group  at 
the  Carle  Clinic  site,  to  five  percent  more  for  the  treatment  group  at  VNS.  Expenditures  for  emer- 
gency room  and  physician  office  visits  were  also  similar  for  treatment  and  control  groups  at  all  sites. 

Table  3.3.4  compares  monthly  inpatient  stays,  emergency  room  visits,  and  physician  office  visits  for 
the  treatment  and  control  groups.  These  measures  of  utilization  were  nearly  the  same  for  treatments, 
and  controls  and  for  the  enrolled  and  the  unenrolled  at  all  CNO  sites. 
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Table  3.3.1 

Medicare  expenditure  per  person  per  month,  72  months  after  random  assignment 


Total  Medicare  expenditures  per  month 

Carle  Clinic-IL     Carondelet-AZ        LAH-MN  VNS-NY 


All  randomized 

beneficiaries                 T          $357*  $479                $416*  $830* 

C          $311  $455                 $364  $765 

CNO  enrollees  vs. 

controls                       T          $356  $461                 $396  $796 

C          $311  $455                 $364  $765 

Services  not  covered  by  CNO 

Carle  Clinic-IL    Carondelet-AZ       LAH-MN  VNS-NY 


All  randomized  benefi- 
ciaries                         T  $274  $376  $332  $674 

C  $275  $396  $334  $664 

CNO  enrollees  vs. 

controls                        T  $265  $357  $308  $625 

C  $275  $396  $334  $664 


CNO-covered  Services 

Carle  Clinic-IL     Carondelet-AZ        LAH-MN  VNS-NY 


All  randomized  benefi- 
ciaries 


$83  $103  $84  $156 

$36  $59  $30  $101 


CNO  enrollees  vs. 

controls  T  $90  $104  $88  $171 

$36  $59  $30  $101 

Note:  All  figures  are  in  1999  dollars.  Means  describe  beneficiaries  randomized  between  Jan  1994  and  Sept  1995  and 
between  Jan  1997  and  Oct  1997.  Total  beneficiary-months  used  for  these  computations  are  shown  in  Appendix 
Table  B.1.  *  indicates  that  treatment/control  differences  in  total  expenditures  are  statistically  significant  at  0.05  level. 
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Table  3.3.2 

Medicare  expenditure  per  person  per  month,  72  months  after  random  assignment 
.  first  six  months  after  random  assignment  and  months  in  1994  not  included  

Total  Medicare  expenditures  per  month 

Carle  Clinic-IL     Carondelet-AZ        LAH-MN  VNS-NY 


All  randomized 

beneficiaries                 T  $361*  $487  ~  $424*  $852 

C  $318  $456  $369  $799 

CNO  enrollees  vs. 

controls                       T  $355  $467  $398  $805 

C  $318  $456  $369  $799 


Services  not  covered  by  CNO 

Carle  Clinic-IL  Carondelet-AZ 


LAH-MN 


VNS-NY 


All  randomized  benefi- 
ciaries 


CNO  enrollees  vs. 
controls 


T 

C 

T 

C 


$278 
$281 

$264 
$281 


$387 
$394 

$367 
$394 


$338 
$338 

$309 
$338 


$695 
$694 

$634 
$694 


CNO-covered  Services 


Carle  Clinic-IL  Carondelet-AZ 


LAH-MN 


VNS-NY 


All  randomized  benefi- 
ciaries 


CNO  enrollees  vs. 
controls 


$84 
$37 


$90 
$37 


$100 
$62 


$100 
$62 


$86 


$31 


$89 
$31 


$158 


$105 


$171 


$105 


Note:      All  figures  are  in  1999  dollars.  Means  describe  beneficiaries  randomized  between  Jan  1994  and  Sept  1995  and 
between  Jan  1997  and  Oct  1997.  Total  beneficiary-months  used  for  these  computations  are  shown  in  Appendix 
Table  B.1 .    indicates  that  treatment/control  differences  in  total  expenditures  are  statistically  significant  at  0.05  level; 
-  indicates  statistically  significant  at  0.10  level. 
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Table  3.3.3 

Medicare  expenditure  per  person  per  month,  72  months  after  random  assignment 
for  inpatlent  care  ,  emergency  room  care,  and  physician  office  visits 


Medicare  inpatient  expenditures  per  month 


Carle  Clinic-IL 

Carondelet-AZ 

LAH-MN 

VNS-NY 

All  randomized 
beneficiaries  T 

$129 

$161 

$181 

$355 

C 

$134 

$165 

$187 

$337 

CNO  enrollees  vs. 
controls  T 

$127 

$154 

$171 

$328 

C 

$134 

$165 

$187 

$337 

Medicare  emergency  room  expenditures  per  month 

Carle  Clinic-IL 

Carondelet-AZ 

LAH-MN 

VNS-NY 

All  randomized  benefi- 
ciaries T 

$1.07 

$1.01 

$1.06 

$1.77 

C 

$0.92 

$0.97 

$0.98 

$1.40 

CNO  enrollees  vs. 

controls  T 

$1.04 

$0.90 

$0.94 

$1.66 

C 

$0.92 

$0.97 

$1.06 

$1.40 

Medicare  physician  office  visits  expenditures  per  month 

Carle  Clinic-IL 

Carondelet-AZ 

LAH-MN 

VNS-NY 

All  randomized  benefi- 
ciaries T 

$14 

$25 

$14 

$37 

$14 

$26 

$14 

$38 

CNO  enrollees  vs. 

controls  T 

$14 

$25 

$14 

$39 

$14 

$26 

$14 

$38 

Note:  All  figures  are  in  1999  dollars.  Means  describe  beneficiaries  randomized  between  Jan  1994  and  Sept  1995  and 
between  Jan  1997  and  Oct  1997.  Total  beneficiary-months  used  for  these  computations  are  shown  in  Appendix 
Table  B.1. 
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Table  3.3.4 

Medicare  utilization  per  person  per  month,  72  months  after  random  assignment 

FOR  INPATIENT  CARE,  EMERGENCY  ROOM  CARE,  AND  PHYSICIAN  OFFICE  VISITS 

Medicare  inpatient  stays  per  month 


Carle  Clinic-IL  Carondelet-AZ 


LAH-MN 


VNS-NY 


All  randomized 
beneficiaries 


CNO  enrollees  vs. 
controls 


T 

C 

T 

C 


0.020 
0.021 

0.019 
0.021 


0.026 
0.028 

0.025 
0.029 


0.027 
0.026 

0.025 
0.026 


0.033 
0.034 

0.030 
0.034 


Medicare  emergency  room  visits  per  month 

Carle  Clinic-IL  Carondelet-AZ 


All  randomized  benefi- 
ciaries 


CNO  enrollees  vs. 
controls 


T 

C 

T 
C 


0.022 
0.020 

0.021 
0.020 


0.025 
0.024 

0.022 
0.024 


LAH-MN 


0.020 
0.020 

0.020 
0.020 


VNS-NY 


0.016 
0.014 

0.014 
0.014 


Medicare  physician  office  visits  per  month 

Carle  Clinic-IL  Carondelet-AZ 


LAH-MN 


VNS-NY 


All  randomized  benefi- 


ciaries 


CNO  enrollees  vs. 
controls 


T 


0.50 
0.51 

0.51 
0.51 


0.77 
0.79 

0.78 
0.79 


0.48 
0.49 

0.49 
0.49 


0.85 
0.86 

0.89 
0.86 


Note:      Means  describe  beneficiaries  randomized  between  Jan  1994  and  Sept  1995  and  between  Jan  1997  and  Oct  1997. 
Total  beneficiary-months  used  for  these  computations  are  shown  in  Appendix  Table  B.1. 
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Table  3.3.5  compares  monthly  expenditures  for  beneficiaries  assigned  to  the  treatment  group  who 
later  dropped  out  with  expenditures  for  those  who  remained  in  the  CNO.  For  purposes  of  the  table,  a 
dropout  was  defined  as  a  beneficiary  who  enrolled  in  the  CNO  and 

•  Remained  in  for  at  least  six  months, 

•  Disenrolled  from  the  CNO, 

•  Was  eligible  for  the  CNO  for  at  least  one  month  after  disenrollment  and 

•  Did  not  later  re-enroll. 

These  last  two  conditions  mean  that  beneficiaries  who  die  will  not  be  considered  as  disenrollees,  nor 
will  those  with  multiple  periods  of  enrollment. 


Table  3.3.5 


Medicare  Expenditures  Per  Person  Per  Month  For  CNO  Dropouts  and  Non-dropouts 


Carle  Clinic-IL 

Carondelet-AZ 

LAH-MN 

VNS-NY 

Non-dropouts 

$323 

$422 

$336 

$660 

CNO  dropouts 

$972 

$998 

$1,595 

$2,547 

For  non-dropouts,  monthly  average  expenditures  are  calculated  over  entire  period  of  CNO  enrollment.  For  dropouts, 
average  expenditures  are  calculated  over  the  six-month  period  prior  to  the  month  of  CNO  dropout. 


Sources:    Abt  Associates  Inc.  CNO  Evaluation  Analytic  Files 


The  results  are  striking.  In  the  six-month  period  prior  to  dropping  out,  those  beneficiaries  who  left  the 
CNO  incurred  expenses  between  two  and  five  times  as  great  as  the  monthly  expenses  of  beneficiaries 
who  remained  in  the  CNO. 

3.4.  Discussion 

In  all  of  the  four  CNO  demonstration  sites,  overall  Medicare  expenditures  per  person  per  month  were 
higher  (by  5  to  15  percent)  for  the  treatment  than  for  the  control  group.  Because  treatment/control 
differences  are  a  downward-biased  estimate  of  CNO  effects,  the  true  net  cost  of  the  CNOs  for  the 
Medicare  program  is  probably  somewhat  larger.  Despite  this  probable  bias,  the  analytic  strength 
provided  by  randomization  justifies  further  investigation  of  this  expenditure  effect. 

The  treatment/control  contrast  indicates  that  mean  monthly  expenditures  for  non-CNO  services  were 
generally  comparable  for  the  two  groups,  though  somewhat  lower  among  the  treatment  group  at  the 
Carondelet  site.  Hence  the  main  impediment  to  achieving  Medicare  budget  neutrality  for  the  CNOs 
was  the  high  level  of  capitation  and  case  management  payments.  These  are  evident  in  the  bottom 
panels  of  Tables  3.3.1  and  3.3.2.  Monthly  expenditures  for  CNO-covered  services  were  higher  for  the 
treatment  than  for  the  control  group  by  50  percent  or  more  in  every  site. 

There  is  no  evidence  that  these  differences  were  biased  either  by  "CNO  startup  effects"  or  by  the 
preponderance  of  early  CNO  months  among  those  in  the  treatment  group.  When  all  data  for  the  first 
year  of  CNO  operations  and  all  data  for  the  first  six  months  after  randomization  were  removed  from 
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the  sample,  the  relative  differences  between  treatment  and  control  groups  in  Medicare  expenditures 
per-person  per-month  changed  little  from  their  values  with  all  data  included. 

We  found  strong  evidence,  by  contrast,  that  beneficiaries  disenrolling  from  the  CNO  had  incurred 
much  higher  Medicare  expenditures  in  the  six  months  prior  to  disenrollment  than  did  other  CNO 
members  who  remained  in  the  CNO  through  the  end  of  data  collection.  There  are  two  possible 
explanations  for  this.  First,  it  is  possible  that  sicker  and  frailer  beneficiaries  who  join  the  CNO  have  a 
higher  risk  of  dropping  out  than  do  other  healthier  members,  perhaps  because  beneficiaries  requiring 
costly  treatment  become  ineligible  for  the  CNO  due  to  the  prohibitions  on  nursing  home  and  hospice 
residency.  Second,  it  is  possible  that  processes  associated  with  higher-than-average  utilization 
encourage  beneficiaries  to  leave  the  CNO.  This  would  not  be  surprising.  Anecdotal  evidence  suggests 
that  many  beneficiaries  who  joined  Medicare  HMOs  during  the  1990s  tended  to  disenroll  when  they 
became  ill  and  wished  to  exercise  more  freedom  of  choice  among  physicians  and  other  providers.  The 
same  mechanism  could  very  well  be  operating  here. 

Regardless  of  the  reason,  these  results  suggest  that  comparing  CNO  enrollees  to  unenrolled  members 
of  the  control  group  is  likely  to  produce  highly  misleading  estimates  of  the  impact  of  the  CNO  on 
Medicare  cost.  If  beneficiaries  who  incur  high  Medicare  expenses  are  more  likely  to  drop  out  of  the 
CNO  than  those  who  do  not,  then  mean  expenditures  for  the  remaining  enrollees  are  naturally  lower 
than  they  would  be  if  they  had  remained  in.  But  there  is  no  mechanism  for  identifying  members  of  the 
control  group  who  would  have  dropped  out.  Hence  comparing  enrollees  to  controls  tends  to 
understate  the  true  net  cost  of  the  CNO  relative  to  fee-for-service  Medicare. 
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4.    Analysis  of  All  Demonstration  Participants 
(Treatment  to  Population  Contrasts) 

The  randomized  design  of  Waves  1  and  3  of  the  CNO  demonstration  (see  Section  1.2.4  for  a 
description  of  demonstration  Waves)  permitted  a  simple  comparison  of  means  between  treatment  and 
control  groups  as  a  measure  of  the  treatment  effect.  The  results  of  these  comparisons  are  presented  in 
Chapter  3.  During  Waves  2  and  4,  however,  applicants  could  enroll  directly  into  the  treatment  group 
and  no  control  group  members  were  assigned.  Consequently,  there  is  no  randomized  control  group  for 
those  enrollees  and  different  methods  are  required  to  measure  the  treatment  effect.  We  chose  a  quasi- 
experimental  approach  whereby  we  constructed  a  comparison  group  based  on  the  fee-for-service 
Medicare  population  in  the  CNO  catchment  area.  However,  we  have  presented  evidence  in  Section 
2.3  that  CNO  applicants  differed  from  the  population  in  the  CNO  catchment  areas  substantially.  The 
proportion  of  disabled  beneficiaries  and  of  minorities  and  women  was  lower  in  the  participant  group, 
and  participants  had  higher  utilization  of  and  spending  for  medical  services.  In  addition,  there  was  a 
higher  proportion  of  beneficiaries  without  any  Medicare  claims  in  the  population  reference  group. 
While  this  does  not  necessarily  mean  that  participants  were  in  worse  health,  one  can  at  least  state  that 
the  two  groups  are  not  comparable.  A  straight  comparison  of  means  would  therefore  reflect  both  a 
treatment  effect  and  a  selection  effect  and  would  provide  a  biased  estimate  of  the  effect  of  CNO 
participation  on  our  spending  and  utilization  measures.  Thus,  adjustment  techniques  that  account  for 
observable  and  unobservable  differences  between  the  treatment  group  and  the  population  reference 
group  were  necessary. 

In  this  chapter,  we  first  show  results  for  the  biased  comparison  of  means,  and  then  introduce  in  a 
stepwise  fashion  a  series  of  statistical  strategies  that  allow  us  to  gradually  disentangle  the  treatment 
and  selection  effects.  Since  the  effect  of  CNO  participation  on  total  cost  per  participant  per  month  is 
the  key  research  question  for  the  evaluation,  we  present  results  for  each  statistical  step  for  this 
subject.  For  brevity  and  to  avoid  possible  confusion,  we  limit  the  presentation  of  results  for  the 
remaining  cost  components  (e.g.,  CNO-services,  inpatient  care)  and  the  utilization  measures 
(physician  visits,  ER  days,  hospital  admissions)  to  the  most  reliable  selection-corrected  model. 

4.1.  Differences  in  Total  Spending 

Differences  in  average  total  Medicare  spending  per  participant  per  month  (PPPM)  are  reported  for  all 
four  sites  in  Table  4.1.1.  The  first  row  displays  the  unadjusted  estimates,  which,  as  discussed  above, 
reflect  both  a  treatment  and  a  selection  effect.  In  all  four  sites,  average  spending  was  significantly 
higher  in  the  treatment  group  than  in  the  population.  The  difference  ranged  from  $228  at  the  VNS  site 
to  $46  at  the  site  with  the  smallest  difference,  Carle  Clinic. 

However,  given  that  CNO  participants  had  higher  baseline  spending  (see  Table  2.3.2),  not  all  of  these 
differences  should  be  attributed  to  CNO  participation.  Adjusted  differences  in  average  monthly 
spending,  accounting  for  observable  beneficiary  risk  factors,  are  presented  in  the  second  row  of  the 
table.  The  risk  adjustment  method,  explained  in  detail  in  Appendix  A,  adjusted  the  spending  estimates 
for  differences  in  age,  sex,  race,  prior  diagnoses  and  prior  expenditures.  Although  the  treatment  group 
still  had  higher  expenditures,  these  adjusted  differences  were  considerably  lower  and  were 
statistically  insignificant  for  two  sites  (Carle  Clinic  and  LAH).  Between  67  percent  (Carondelet)  and 
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72  percent  (Carle  Clinic)  of  the  unadjusted  differences  could  be  explained  by  differences  in  casemix, 
demonstrating  the  power  and  importance  of  the  risk  adjustment  model. 

In  addition  to  observable  factors  of  beneficiary  risk,  we  also  adjusted  for  the  impact  of  random  events 
that  occurred  in  one  or  more  years  of  the  observation  period,  such  as  a  flu  epidemic,  by  using  so- 
called  year  fixed  effects.  The  third  row  of  Table  4.1.1  illustrates  that  adjusting  for  year  fixed  effects 
had  almost  no  effect  on  the  estimated  differences  between  spending  for  treatment  group  members  and 
spending  for  the  population.  This  is  not  surprising,  since  it  is  unlikely  that  such  events  would  have 
affected  one  of  the  two  groups  more  than  the  other. 

Table  4.1.1 

Estimated  Differences  in  Per  Month  Spending  between  CNO  Treatment  Group  and  the 
Population  Reference  Group  for  Ail  Services  in  All  Years  (1994-1999) 

Carle  Carondelet  LAH  VNS 


Unadjusted                          $46  ***  $91  ***                $52  **  $228  *** 

Adjusted  for  Risk                    $9  $30   **                 $14  $66  ** 

Adjusted  for  Risk                    $8  $29   **                 $15  $66  ** 

and  Year  Effects 

All  dollar  amounts  were  converted  to  1 999  constant  dollars.  The  CNO  treatment  group  consists  of  all  those 
randomized  to  treatment  or  directly  enrolled  in  the  CNO.  *  denotes  significance  at  p<0.05;  **  p<0.0 1 ;  ***p<0.00 1 . 

Sources:    Abt  Associates  Inc.  CNO  Evaluation  Analytic  Files 


4.2.  Differences  in  Spending  Changes  Over  Time 

Despite  the  predictive  power  of  our  risk  adjustment  method,  it  is  possible  that  part  of  the  remaining 
spending  differences  might  be  traceable  to  unaccounted  for  baseline  differences  between  CNO 
participants  and  the  population.  The  risk  adjustment  models  can  only  correct  for  differences  that  are 
observable  on  the  basis  of  the  administrative  data  available  to  the  study,  such  as  demographic 
characteristics  or  prior  diagnoses.  Information  was  not  available  on  other  characteristics,  such  as 
beneficiary  preferences  or  current  health  status,  and  it  is  reasonable  to  think  that  those  factors  may 
have  influenced  both  the  decision  to  join  a  CNO  and  the  level  of  Medicare  spending. 

To  shed  further  light  on  the  question  of  whether  CNO  participation  had  an  effect  on  Medicare 
spending,  we  examined  differences  in  spending  changes  over  time.  In  other  words,  starting  from  the 
baseline  year  of  1994  (the  first  year  of  the  demonstration)  did  average  spending  change  differently  for 
participants  and  the  population?  For  example,  assume  that  average  spending  in  1994  was  $60  per 
treatment  group  member  and  $45  in  the  population,  and  that  spending  in  1995  rose  to  $90  in  the 
treatment  and  $65  in  the  reference  group.  Thus,  the  change  in  spending  was  $30  per  month  in  the 
treatment  group  compared  to  $20  in  the  population,  and  the  differential  change  was  $10.  This  would 
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indicate  that  cost  increased  by  $10  more  in  the  treatment  group  in  that  year,  suggesting  that  the 
demonstration  caused  costs  to  increase  faster  than  they  would  have  otherwise. 

This  approach  is  referred  to  as  the  difference-in-differences  method,  since  it  investigates  differences 
in  changes  over  time  for  two  groups  to  be  compared.  The  advantage  of  this  approach  is  that  baseline 
differences,  which  may  be  a  consequence  of  non-random  selection,  are  not  considered  and  only 
differences  in  the  change  from  that  baseline  are  analyzed.  Thus,  unobservable  differences  between  the 
two  groups  that  are  present  at  baseline  do  not  bias  estimated  effects.  By  analyzing  changes  over  the 
years  of  the  demonstration,  we  can  also  address  the  question  of  whether  the  sites  became  more 
efficient  over  time.  The  underlying  hypothesis  is  that  CNO  sites  might  not  have  been  cost-effective  in 
the  early  years  since  enrollees  had  not  had  time  to  benefit  from  preventive  services  and  the  sites  were 
still  improving  their  model  of  care,  but  that  later  years  would  show  more  positive  health  effects.  The 
results  of  this  analysis  are  summarized  in  Table  4.2. 1 . 


Table  4.2.1 


Estimated  Differences  between  the  CNO  Treatment  Group  and  the  Population  Reference 
Group:  Changes  in  Per  Month  Spending  for  All  Services,  Relative  to  1994 


Year 

Carle 

Carondelet 

LAH 

VNS 

1995 

-$1 

$26 

$121  *** 

$153  ** 

1996 

$3 

$3 

$116  ** 

$153  * 

1997 

$22- 

$45 

$111  ** 

$277  *** 

1998 

$11 

$45 

$101  * 

$377  *** 

1999 

$60  ** 

$75  * 

$192  *** 

$373  *** 

All  dollar  amounts  were  converted  to  1 999  constant  dollars.  All  differences  are  adjusted  for  beneficiary  risk  and  year 
effects.  The  CNO  treatment  group  consists  of  all  those  randomized  to  treatment  or  directly  enrolled  in  the  CNO. 
Sample  sizes  are  described  in  Appendix  Table  C.  1 .  ~  denotes  significance  at  p<0. 1 ;  *  p<0.05;  **  p<0.0 1 ;  ***p<0.00 1 . 


Sources:    Abt  Associates  Inc.  CNO  Evaluation  Analytic  Files 


The  results  clearly  demonstrate  that  average  monthly  Medicare  spending  increased  much  faster  in  the 
treatment  group  than  in  the  population.  In  two  sites  (LAH  and  VNS),  the  difference  in  changes  was 
already  statistically  significant  in  1995,  whereas  it  became  significant  only  in  the  last  year  at  the  Carle 
Clinic  and  Carondelet  sites.  All  four  sites  showed  a  steady  increase  in  estimated  differences  through 
time,  indicating  that  average  spending  in  the  treatment  group  kept  increasing  relative  to  the 
population  over  the  course  of  the  demonstration.  Not  only  are  the  estimates  statistically  significant, 
but  also  of  a  substantial  magnitude:  average  spending  increased  by  as  much  as  $377  more  per  CNO 
participant  per  month  in  the  New  York  site,  and  $60  more  in  the  site  with  the  smallest  differences, 
Carle  Clinic. 

The  same  methodology  was  used  to  assess  the  hypothesis  that  a  beneficiary  had  to  be  enrolled  in  a 
CNO  over  a  certain  period  for  the  positive  effects  of  case  management  on  cost  to  materialize.  For  this 
analysis,  we  removed  those  treatment  group  members  who  had  less  than  six  months  of  enrollment.  As 
shown  in  Appendix  Table  C.  1,  this  definition  of  CNO  treatment  selectively  removes  about  a  third  of 
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the  person-years  from  the  treatment  group,  while  the  population  reference  group  remains  unchanged. 
The  results  are  summarized  in  Table  4.2.2.  Although  substantially  lower  on  average,  the  estimates 
show  the  same  pattern  as  the  ones  presented  above:  differences  in  monthly  spending  between  CNO 
participants  and  the  reference  group  grew  over  time,  with  the  exception  that  the  differences  at  Carle 
Clinic  were  not  statistically  significant  in  any  year.  This  reduction  in  differences  may  have  been  the 
effect  of  CNO  services,  or,  as  discussed  in  Section  3.4,  of  costly  enrollees  choosing  to  disenroll. 

There  also  was  a  temporary  drop  in  the  differences  in  1998,  possibly  reflecting  the  impact  of  the 
Balanced  Budget  Act  of  1997.12  This  drop  was  to  some  degree  noticeable  in  Table  4.2.1  but  is  much 
more  marked  here.  These  findings  confirm  that  CNO  enrollment  led  to  higher  Medicare  spending 
than  under  fee-for-service,  and  that  this  differential  increased  over  the  course  of  the  demonstration. 


Table  4.2.2 


Estimated  Differences  between  the  CNO  Treatment  Group  Enrolled  at  Least  Six  Months  and 
the  Population  Reference  Group:  Changes  in  Per  Month  Spending  for  All  Services,  Relative 
to  1994 


Year 

Carle 

Carondelet 

LAH 

VNS 

1995 

$2 

$63 

$74 

$172  * 

1996 

-$2 

$81  ~ 

$70 

$152- 

1997 

$6 

$112  * 

$54 

$225  ** 

1998 

-$20 

$68 

$40 

$211  * 

1999 

$35 

$103  * 

$133- 

$275  ** 

All  dollar  amounts  were  converted  to  1999  constant  dollars.  All  differences  are  adjusted  for  beneficiary  risk  and  year 
effects.  The  CNO  treatment  group  consists  of  all  those  randomized  to  treatment  or  directly  enrolled  in  the  CNO 
provided  they  were  enrolled  for  at  least  six  months.  ~  denotes  significance  at  p<0. 1 ;  *  p<0.05;  **  p<0.0 1 ;  ***p<0.00 1 . 
Sample  sizes  are  described  in  Appendix  Table  C.  1 . 


Sources:    Abt  Associates  Inc.  CNO  Evaluation  Analytic  Files 


In  summary,  these  results  suggest  that  the  CNO  as  a  model  of  care  provision  was  associated  with 
increased  cost  to  the  Medicare  program  compared  to  traditional  fee-for-service  payment.  This 
conclusion  is  based  on  very  robust  findings  that  were  consistent  across  several  analytic  approaches. 
The  differences  persisted  after  the  application  of  increasingly  complex  risk  adjustment  methods  so 
one  can  be  confident  that  they  were  not  due  to  baseline  differences  between  the  treatment  group  and 
the  fee-for-service  population.  In  addition,  these  differences  were  robust  to  changes  in  the  way  CNO 
participation  was  defined.  While  the  cost  differential  was  consistently  smallest  at  the  Carle  Clinic  site 
and  largest  at  the  VNS  site,  it  increased  over  time  at  all  four  sites,  with  the  exception  of  1998, 
possibly  reflecting  an  effect  of  the  Balanced  Budget  Act  of  1997. 


12    The  Balanced  Budget  Act  of  1 997  substantially  reduced  Medicare  outlays  for  home  health  and  other 
services.  Several  of  its  most  restrictive  provisions,  including  those  affecting  home  health  providers,  were 
changed  after  1998  in  response  to  provider  concerns. 
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4.3.  Decomposition  of  Differences  in  Total  Spending 


Having  established  that  CNO  participation  was  associated  with  higher  average  monthly  Medicare 
expenditures,  we  analyzed  which  components  of  spending  were  responsible  for  this  difference.  We 
again  used  the  risk-adjusted  difference-in-differences  methodology,  as  described  above,  to  account 
for  observable  and  unobservable  characteristics  other  than  CNO  participation  that  might  have 
contributed  to  differences  in  spending.  The  first  step  was  to  decompose  total  Medicare  spending  into 
spending  for  services  provided  under  the  CNO  arrangements  ("in-bundle  services,"  see  Section  1.1.3 
for  a  detailed  definition),  which  corresponds  to  the  capitation  and  case  management  fees  for  the 
treatment  group,  and  all  other  services  ("out-of-bundle  services").  These  results  are  presented  in 
Table  4.3.1. 


Table  4.3.1 

Estimated  Differences  between  the  CNO  Treatment  Group  and  the  Population  Reference 
Group:  Changes  in  Per  Month  Spending  for  In-bundle  and  Out-of-bundle  Services,  Relative  to 
1994 


Carle 

Carondelet 

LAH 

VNS 

In-Bundle 

1995 

$1 

$17  ** 

$31  *** 

$21  * 

•  1996 

$15  *** 

-$1 

$18  * 

$17- 

1997 

$14  ** 

$11 

$33  *** 

$55  *** 

1998 

$12  * 

$22  ** 

$36  *** 

$105  *** 

1999 

$0 

$14- 

$52  *** 

$96  *** 

Out-of-Bundle 

1995 

-$1 

$3 

$87  ** 

$128  * 

1996 

$3 

-$9 

$79  * 

$128  * 

1997 

$22 

$22 

$67  * 

$201  *** 

1998 

$11 

$12 

$62 

$267  *** 

1999 

$60  * 

$42 

$142  ** 

$255  *** 

All  dollar  amounts  were  converted  to  1999  constant  dollars.  All  differences  are  adjusted  for  beneficiary  risk  and  year 
effects.  The  CNO  treatment  group  consists  of  all  those  randomized  to  treatment  or  directly  enrolled  in  the  CNO. 
Sample  sizes  are  described  in  Appendix  Tabled,  -  denotes  significance  at  p<0. 1 ;  *p<0.05;  **p<0.01; 
***p<0.001 .  Note  that,  since  in-bundle  and  out-of-bundle  components  were  estimated  in  separate  statistical  models, 
they  do  not  necessarily  add  up  to  the  differences  in  total  spending  presented  above. 

Sources:    A  bt  Associates  Inc.  CNO  Evaluation  A  nalytic  Files 


For  CNO-covered  services,  the  estimates  show  that  average  monthly  cost  in  the  treatment  group 
increased  substantially  faster  than  in  the  population,  and  did  so  steadily  over  time.  The  effect  is  more 
marked  at  the  LAH  and  VNS  sites,  where  this  difference  was  statistically  significant  in  every  single 
year  and  of  much  higher  magnitude  than  at  the  two  other  sites.  However,  even  at  Carle  and 
Carondelet,  demonstration  costs  increased  significantly  faster  than  population  costs  in  three  of  the 
five  years.  These  results  confirm  our  conclusion  from  Chapter  3  that  payment  rates  for  the  CNO  sites 
were  set  too  high  to  achieve  budget  neutrality. 
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By  contrast,  the  results  for  non-CNO  services  do  not  match  up  with  the  findings  from  the  analysis  of 
the  randomized  portion  of  the  demonstration.  When  comparing  CNO  applicants  randomly  assigned  to 
treatment  or  control  status,  we  found  that  average  spending  for  those  services  was  similar  in  both 
groups.  In  contrast,  the  results  from  the  entire  demonstration  suggest  that  non-CNO  spending 
increased  disproportionately  for  CNO  participants  over  time.  To  shed  further  light  on  this 
discrepancy,  we  examined  spending  for  three  major  components  of  non-CNO  spending:  inpatient 
care,  ER  visits,  and  physician  office  visits.  These  results  are  summarized  in  Table  4.3.2. 


Table  4.3.2 


Estimated  Differences  between  the  CNO  Treatment  Group  and  the  Population  Reference 
Group:  Changes  in  Per  Month  Spending  for  Inpatient  Hospital  Services,  ER  Visits  and 
Physician  Office  Visits,  Relative  to  1994 


Carle 

Carondelet 

LAH 

VNS 

Hospital  Services 

1995 

$3 

$5 

$55  * 

(POO 

$82  ~ 

1996 

$2 

-$15 

$58  * 

$104  * 

1997 

$12 

$12 

$42 

$132  ** 

1998 

$0 

$5 

$42 

$175  *** 

1999 

$43  ~ 

$14 

$104  * 

$193  ** 

ER  Visits 

1995 

$0.28 

$0.6 1 

$0.11 

$0.42 

1996 

$0.44  ~ 

$0.17 

$0.13 

$0.45 

1997 

$0.12 

$0.15 

-$0.01 

$0.40  ~ 

1998 

$0.44  * 

$0.13 

$0.04 

$0.03 

1999 

$0.35  * 

$0.19 

$0.02 

$0.20 

Physician  Office  Visits 

1995 

-$0.60  ~ 

$1.04 

$0.63 

-$1.18 

1996 

$0.72  * 

-$0.42 

$0.15 

$0.94 

1997 

$0.40 

$0.43 

$0.82  ~ 

$0.80 

1998 

$0.32 

$0.03 

$0.51 

-$0.42 

1999 

$0.30 

$0.83 

-$0.78 

-$0.96 

All  dollar  amounts  were  converted  to  1999  constant  dollars.  All  differences  are  adjusted  for  beneficiary  risk  and  year 
effects.  The  CNO  treatment  group  consists  of  all  those  randomized  to  treatment  or  directly  enrolled  in  the  CNO. 
Sample  sizes  are  described  in  Appendix  Table  C.  1 .  ~  denotes  significance  at  p<0. 1 ;  *  p<0.05;  **  p<0.01 ; 
***p<0.00 1 .  Note  that  these  spending  components  are  not  exhaustive  and  were  estimated  in  separate  statistical 
models;  they  do  not  necessarily  add  up  to  the  differences  in  total  out-of-bundle  spending  presented  above. 


Sources:    Abt  Associates  Inc.  CNO  Evaluation  Analytic  Files 


These  findings  suggest  that  the  more  rapid  cost  increase  in  the  treatment  group  can  mainly  be 
attributed  to  hospital  services.  This  comes  as  no  surprise,  given  that  hospital  services  were  by  far  the 
largest  component  of  monthly  Medicare  costs  (see  Table  3.3.3).  In  particular  in  LAH  and  VNS,  the 
two  sites  with  the  largest  cost  differentials,  but  also  in  the  two  other  sites,  average  spending  for 
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hospital  services  increased  substantially  more  in  the  treatment  group.  In  addition,  cost  for  ER  visits 
increased  disproportionately  for  the  Carle  Clinic  enrollees,  whereas  changes  in  spending  for  physician 
office  visits  showed  no  differential  pattern.  Further  support  for  these  findings  can  be  derived  from  the 
utilization  analysis  in  which  we  compared  changes  in  the  number  of  physician  office  visits,  as  well  as 
in  the  probability  of  an  ER  visit  and  of  a  hospital  admission,  as  displayed  in  Table  4.3.3. 


Table  4.3.3 


Estimated  Differences  between  the  CNO  Treatment  Group  and  the  Population  Reference 
Group:  Changes  in  Per  Month  Utilization  of  Inpatient  Hospital  Admissions,  ER  Visits  and 
Physician  Office  Visits,  Relative  to  1994 


Carle 

Carondelet 

LAH 

VNS 

Hospital  Admissions 

1995 

-0.03 

-0.13 

0.20  ~ 

A    1  A 

0.10 

1996 

0.16  ~ 

-0.14 

0.22  * 

A  1  c\  * 

0.19 

1997 

0.15  ~ 

-0.05 

0.18  ~ 

A  T7  ** 

0.2/ 

1998 

0.07 

-0.06 

0.16 

0.41  *** 

1999 

a  i  a 
0.10 

u.uo 

0  39  *** 

ER  Visits 

1995 

-0.03 

0.00 

0.33  ** 

-0.03 

1996 

0.04 

-0.05 

0.24* 

0.07 

1997 

-0.01 

0.01 

0.28  * 

0.09 

1998 

-0.05 

-0.12 

0.20- 

0.17 

1999 

0.01 

-0.03 

0.14 

0.07 

Physician  Office  Visits 

1995 

-0.0-1 

0.01 

0.02 

-0.03 

1996 

0.03  ** 

-0.03 

0.01 

0.01 

1997 

0.02 

0.01 

0.03  * 

0.02 

1998 

0.02 

-0.01 

0.01 

0.00 

1999 

0.00 

0.01 

-0.01 

-0.02 

All  differences  are  adjusted  for  beneficiary  risk  and  year  effects.  The  CNO  treatment  group  consists  of  all  those 
randomized  to  treatment  or  directly  enrolled  in  the  CNO.  ~  denotes  significance  at  p<0. 1 ;  *  p<0.05;  **  p<0.01 ; 
***p<0.001 .  Sample  sizes  are  described  in  Appendix  Table  C.  1 . 


Sources:    Abt  Associates  Inc.  CNO  Evaluation  Analytic  Files 


The  probability  of  a  hospital  admission  increased  more  for  beneficiaries  enrolled  in  the  LAH  and 
VNS  sites  than  for  the  fee-for-service  population  in  the  corresponding  catchment  areas.  There  is  also 
evidence  of  higher  hospital  utilization  for  the  Carle  Clinic  treatment  group,  whereas  there  was  no 
difference  in  hospital  utilization  for  Carondelet  enrollees,  the  site  that  also  had  the  smallest 
differentials  in  total  non-CNO  and  hospital  services  spending.  With  the  exception  of  elevated  ER 
utilization  at  LAH,  the  results  for  physician  and  ER  visits  were  generally  not  significant  and  showed 
no  clear  pattern. 
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If  one  were  to  interpret  these  results  without  the  context  provided  by  the  findings  from  the 
randomized  portion  of  the  demonstration,  the  conclusion  would  be  unambiguous:  care  for  enrollees 
was  shifted  to  the  inpatient  setting,  and  the  magnitude  of  this  shift  increased  over  time.  This  was 
particularly  true  at  the  LAH  and  VNS  sites,  and  to  a  lesser  degree  at  Carle  Clinic,  and  is  consistent 
with  the  incentive  structure  that  the  CNO  demonstration  created:  In  any  payment  arrangement  in 
which  the  provider  bears  the  burden  of  the  marginal  cost  of  selected  areas  of  care  there  is  a  strong 
incentive  to  avoid  cost  by  shifting  the  provision  of  care  away  from  those  areas.  Since  the  CNOs 
would  not  have  to  provide  any  care  to  a  beneficiary  while  hospitalized,  but  would  still  collect  the 
monthly  payments,  hospital  care  appears  to  have  been  a  particularly  appealing  option  for  cost 
shifting. 

The  main  challenge  to  this  interpretation  is  the  fact  that  a  comparison  of  average  spending  of 
demonstration  applicants  randomized  to  treatment  or  control  status  did  not  show  any  meaningful 
differences  in  non-CNO  or  hospital  spending.  Two  explanations  for  this  discrepancy  can  be 
hypothesized.  First,  no  risk  adjustment  method  can  be  as  powerful  as  true  random  assignment.  To  be 
able  to  extend  the  evaluation  to  the  later,  non-randomized  portion  of  the  demonstration,  we  had  to  use 
a  quasi-experimental  design  in  which  the  fee-for-service  Medicare  population  living  in  a  CNO 
catchment  area  was  used  as  comparison  group.  As  demonstrated  in  Section  2.3,  this  group  differs  in 
many  important  respects  from  the  demonstration  participants.  While  we  have  tried  to  account  for 
those  differences  with  well-established  and  powerful  methods,  there  will  always  be  a  residual 
possibility  that  self-selection  into  the  demonstration  has  biased  the  results.  However,  given  the  risk- 
adjusted  difference-in-differences  approach,  one  would  have  to  postulate  a  selection  process  that  had 
a  differential  impact  over  the  course  of  the  demonstration,  and  we  have  not  identified  a  plausible 
hypothesis  for  such  a  process. 

The  second  hypothesis  is  that  the  shifting  of  care  to  the  inpatient  sector  was  a  provider  effect  rather 
than  an  incentive  effect.  In  other  words,  staff  at  LAH  and  VNS,  and  to  a  lesser  degree  at  Carle  Clinic, 
were  more  inclined  to  send  beneficiaries  to  the  hospital  than  the  average  health  care  provider  for  the 
Medicare  population.13  They  would  do  so  regardless  of  treatment  or  control  status,  as  this  was  just 
their  prevailing  pattern  of  care  provision.  This  is  not  meant  to  imply  that  those  additional 
hospitalizations  were  strictly  unnecessary,  but  only  that  a  beneficiary  with  contact  with  one  of  these 
three  CNOs  was  more  likely  to  be  hospitalized  with  a  marginal  indication.  The  provider  effect 
hypothesis  is  also  consistent  with  the  temporal  pattern  observed  in  the  data.  During  the  demonstration 
period,  there  was  a  strong  secular  trend  in  the  Medicare  population  towards  fewer  hospitalizations  and 
shorter  hospital  stays,  resulting  in  substantial  shifting  from  the  inpatient  to  the  outpatient  sector.  If 
site  staff  maintained  their  established  practice  patterns  despite  this  trend,  their  clientele  would 
increasingly  have  higher  hospital  spending  than  the  population.  We  will  further  investigate  this 
hypothesis  in  preparation  for  our  Final  Report  to  Congress. 


Since  the  CNO  nurse  case  managers  could  not  directly  admit  patients,  one  also  has  to  assume  that  these 
sites  worked  with  physicians  who  were  receptive  to  their  recommendations. 
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5.    Next  Steps 


This  Preliminary  Report  to  Congress  is  the  first  of  two  to  be  produced  by  the  Phase  n  Evaluation  of 
the  Community  Nursing  Organization.  Analysis  for  the  Final  Report  will  be  conducted  in  the  fall  of 
2001  and  will  depart  from  the  analysis  in  this  report  in  three  important  ways.  First,  an  additional  year 
of  demonstration  data  will  be  available  (calendar  year  2000),  so  some  of  the  analyses  reported  in 
Chapters  3  and  4  of  this  report  will  be  repeated  including  the  new  data.  Second,  the  project  team  will 
analyze  measures  of  mortality  and  the  incidence  of  potentially  avoidable  hospitalizations  in  an 
attempt  to  detect  any  effect  of  participation  in  the  CNO  on  quality  of  care.  Finally,  we  will  further 
investigate  the  influence  of  self-selection  into  the  demonstration  in  at  least  two  ways: 

1)  An  exploration  of  possible  provider  effects  on  the  patterns  of  care  received  by  both 
treatments  and  controls,  and 

2)  The  development  of  an  explicit  statistical  model  of  the  application  decision  that  can  be  linked 
to  analysis  of  outcomes  to  more  explicitly  account  for  possible  selection  bias. 

The  first  task  will  involve  an  exploration  of  hospitalization  trends  in  the  population  reference  group 
compared  to  trends  in  both  treatment  and  control  groups  at  the  LAH  and  VNS  sites.  The  second  task 
will  focus  exclusively  on  the  treatment  and  population  reference  groups  to  build  a  statistical  model  of 
selection.  This  two-part  statistical  model,  known  as  a  "switching  regression  model,"  is  the  most 
powerful  method  available  to  researchers  faced  with  non-random  selection,  but  estimating  it  requires 
substantial  computational  and  analytical  resources.  Due  to  this  complexity,  we  will  perform  the 
switching  analysis  only  once  for  this  evaluation,  when  data  for  the  entire  study  period  becomes 
available. 

Finally,  the  Phase  II  Final  Report  will  also  include  results  from  the  sites'  survey  of  beneficiary 
satisfaction. 
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Appendix  A:  Detailed  Methodology 


This  technical  appendix  provides  additional  detail  on  the  analytic  methods  employed  to  produce  the 
results  in  this  report.  As  in  past  CNO  evaluation  reports,  our  first  approach  compared  mean  (average) 
values  for  members  of  the  treatment  group  to  mean  values  for  members  of  the  control  group,  where 
such  a  comparison  was  possible.  However,  since  randomization  to  treatment  and  control  groups  was 
not  consistently  implemented  in  the  latter  years  of  the  demonstration  (see  Section  1.2.3),  more 
complex  methods  became  necessary.  We  implemented  two  such  approaches:  comparison  of  risk- 
adjusted  means  and  calculation  of  difference-in-differences  estimators. 

Although  for  ease  of  exposition  the  following  discussion  generally  refers  to  total  Medicare 
expenditures  as  the  outcome  of  interest,  each  of  these  methods  were  applied  to  a  series  of  variables  of 
interest,  producing  the  results  reported  in  Chapters  3  and  4.  The  next  section  defines  these  variables 
in  detail. 

Description  of  Outcome  Variables  to  be  Analyzed 

The  project  team  evaluated  the  effect  of  CNO  participation  on  Medicare  expenditures  and  utilization 
of  care.  The  following  expenditure  variables  were  used  for  the  treatment,  control,  and  population 
reference  groups.  All  costs  were  expressed  in  payments  per  person  per  month  (PPPM). 

•  .  Adjusted  CNO-covered  expenditures:  This  variable  accounts  for  claims  for  services  covered 

according  to  the  definitions  in  Section  1.1.3  as  well  as  CNO-capitation  and  case  management 
payments.  As  there  was  no  lockout  mechanism  in  place,  treatment  group  members  could  still 
obtain  services  on  a  fee-for-service  basis.  CMS  managed  this  situation  by  recovering  from  the 
CNO  the  capitation  payment  for  each  CNO  member  for  those  months  in  which  he/she  received 
more  than  $120  in  CNO  services  outside  the  CNO,  i.e.  under  fee-for-service  ($100  in  1995).  The 
adjusted  expenditure  variables  reflect  this  payment  correction. 

•  Non-CNO  expenditures:  This  variable  accounts  for  claims  for  all  services  not  included  in  the 
CNO-package,  as  defined  in  Section  1.1.3.  Beyond  the  overall  total,  subtotals  of  claims  for 
inpatient  hospital  services,  emergency  room  services,  and  physician  office  visits  were  also 
analyzed. 

•  Total  expenditures:  This  variable  contains  the  sum  of  all  costs  to  the  Medicare  program,  including 
claims,  capitation  payments,  and  case  management  fees.  This  variable  is  the  sum  of  adjusted 
CNO-covered  expenditures  and  non-CNO  expenditures. 

Utilization  for  each  beneficiary  was  measured  in  three  categories: 

•  Inpatient  admissions:  Number  of  hospital  admissions 

•  Emergency  room  visits:  Number  of  days  with  a  claim  for  ER  visits 

•  Physician  visits:  Number  of  claims  for  outpatient  physician  office  visits  or  office  consultations 

These  categories  were  chosen  to  correspond  with  the  subcategories  of  non-CNO  expenditures  to 
facilitate  the  investigation  of  whether  and  to  what  degree  CNO-enrollment  resulted  in  cost-neutral 
changes  in  utilization  patterns. 


Abt  Associates  Inc. 


Phase  II  Evaluation  of  CNO  Demonstration 


55 


Analysis  of  Randomized  Participants  (Treatment  to  Control 
Contrasts) 


We  computed  two  separate  cross-section  contrasts  of  treatment  and  control  groups  that  mirror  those 
included  in  the  CNO  Final  Evaluation  Report  (Abt  Associates,  2000).  The  first  of  these  was  the  so- 
called  intent-to-treat  (ITT)  contrast  that  assigns  each  person  to  the  group  to  which  he  or  she  was 
randomized  regardless  of  subsequent  enrollment  or  dropout  behavior.  While  the  enrollment  into  the 
CNO  of  some  700  beneficiaries  originally  assigned  to  the  control  group  almost  inevitably  biased  this 
comparison,  it  formed  a  useful  contrast  with  its  natural  counterpart,  which  we  termed  the  enrolled- 
treatments-vs-controls  (ETC)  contrast.  That  counterpart,  also  biased,  defined  the  treatment  group  as 
those  individuals  randomized  to  treatment  who  actually  enrolled  in  the  CNO.  They  remained  in  the 
treatment  group  only  so  long  as  they  remained  enrolled  in  the  CNO.  The  control  group  was  defined  as 
those  persons  who  were  assigned  to  the  control  group  but  did  not  subsequently  enroll  in  the  CNO.  For 
both  the  ITT  and  ETC  contrasts,  individuals  remained  in  the  sample  only  for  those  months  when  they 
were  alive  and  believed  to  be  eligible  for  enrollment  in  the  CNO  (residing  in  the  CNO  catchment 
area,  enrolled  in  Part  A  and  B,  not  receiving  hospice  benefits,  and  not  enrolled  in  a  Medicare  HMO  or 
health  care  prepayment  plan). 

At  this  point,  ITT  and  ETC  both  lead  to  biased  and  different  estimates  of  CNO  effects  -  ITT  because 
the  control  group  has  become  contaminated  by  enrollment  of  some  controls  in  the  CNO,  and  ETC  by 
contamination  and  the  self  selection  of  members  of  the  treatment  group  into  or  out  of  the  CNO. 
Nevertheless  there  was  some  reason  to  believe  that  the  biases  of  ITT  and  ETC  were  in  opposite 
directions.  ITT  might  understate  CNO  effects  (because  some  ITT  controls  received  CNO  services) 
and  ETC  might  overstate  them  (because  those  with  the  least  to  gain  from  the  CNO  might  be  less 
likely  to  enroll  and  more  likely  to  drop  out). 

For  each  individual  randomized  to  the  treatment  or  control  group,  total  expenditures  for  each  type  of 
service  were  calculated  for  each  month  after  randomization.  These  were  summed  over  all  months 
during  which  the  individual  was  eligible  for  the  CNO  from  the  time  of  randomization  to  a  specified 
follow-up  date  up  to  72  months  later.  Total  expenditures  over  each  group  were  then  divided  by  total 
eligible  person-months  to  compute  expenditures  per-person  per-month  (PPPM).  The  calculations  are 
illustrated  in  Equation  [1]  below.  Let 

E\]n  E2Tit ,  EX^ ,  E2CU  =  deflated  expenditures'4  for  CNO-type  services  (El)  and  non-CNO 

services  (E2)  in  month  t  after  randomization  by  individuals  i  assigned  to 
the  treatment  and  control  groups  respectively; 

dT  dc  =  1   if  individual  i,  assigned  to  either  treatment  or  control  group 

"it  '      it  '  °^ 

respectively,  was  eligible  for  the  CNO  in  month  t  after  randomization; 
=  0  otherwise; 

d?  =  1  if  d]t  =1  and  individual  i  assigned  to  the  treatment  group  was  enrolled  in  the 

CNO  in  month  t; 
=  0  otherwise. 

14    Expenditures  reflect  claims  for  Medicare-covered  services,  as  well  as  capitation  and  case  management 
payments  to  the  CNO  sites.  As  outlined  in  Section  1 .3. 1 ,  the  total  expenditure  variable  reflects  adjustments 
to  capitation  payments  for  out-of-plan  use  of  services. 
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Then  expenditures  for  CNO  services  PPPM  under  the  intent-to-treat  model  from  time  of 
randomization  until  month  M  after  randomization  for  the  treatment  and  control  groups  are  given  by 

E\TM  and  ElcM  in  Equation  [1]  below: 


[i] 


El 


i  (=1 


M 


M 

Z2X 


(  /=1 


M 

I2X 

i  t=\ 


The  numerators  in  [1]  are  just  the  sum  over  persons  and  months  of  expenditures  for  CNO  services  for 
those  months  when  randomized  individuals  were  eligible  to  be  enrolled  in  the  CNO.  The 
denominators  are  simply  the  total  number  of  person-months  of  eligibility  over  the  selected  M  months 
from  randomization.  It  should  be  immediately  clear  that  expenditures  for  non-CNO  services  PPPM 
( E2TM  and  E2CM  )  over  the  period  from  randomization  to  month  M  are  computed  in  the  same  way 
merely  by  replacing  Ell  and  Elf,  with  E2Tit  and  E2cit  in[l]. 

The  ETC  comparison  was  computed  in  a  similar  manner.  The  comparison  of  currently  enrolled 
treatments  with  members  of  the  control  group  was  carried  out  by  replacing  dit  in  [1]  with  du  . 
Treatment  group  expenditures  PPPM  ( ElTM  and  E2TM  )  are  given  by 

M 

[2]  ElTM=^^  


E2^  = 


i  t=l 


i  t=l 


And  since  the  composition  of  the  control  group  is  unaffected  by  the  change  from  ITT  to  >ETC, 
expenditures  PPPM  for  the  control  group  are  the  same  as  in  [1]. 

Equations  [1]  and  [2]  will  support  further  analysis  of  CNO  effects  on  long-term  enrollees  as  well.  By 
raising  the  lower  limit  of  t  from  1  to  6,  [1]  and  [2]  produce  comparisons  that  are  limited  to  person- 
months  beyond  the  first  six  months  after  randomization. 

Analysis  of  All  Participants  (Treatment  to  Population  Contrasts) 

When  the  above  analyses  were  restricted  to  comparisons  of  "intent  to  treat"  treatments  and  controls 
through  October  1997,  the  randomized  design  of  the  demonstration  ensured  a  fair  comparison. 
Unfortunately  (for  the  evaluation),  the  periods  of  open  enrollment  and  the  decision  to  allow  controls 
to  switch  to  the  treatment  group  after  October  1997  meant  that  substantial  periods  of  time  and 
numbers  of  participants  were  without  a  properly  randomized  control  group.  Evaluating  the  effects  of 
the  demonstration  during  these  periods  and  on  these  participants  required  addressing  the  problem  of 
selection  bias. 
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In  what  follows,  we  discuss  estimation  techniques  that  were  applied  to  the  problem  of  self-selection 
or  selection  bias  that  arose  in  the  CNO  demonstration.  To  simplify  the  discussion  we  mostly  neglect 
the  existence  of  the  control  group,  focusing  attention  on  the  contrast  between  the  treatment  group 
(participants)  and  the  population  reference  group  (non-participants).  As  we  implemented  these 
techniques,  however,  we  took  advantage  of  the  pre-October  1997  control  group  to  test  the  efficacy  of 
each  technique,  contrasting  the  control  group  with  the  selection-corrected  population  reference  group 
to  determine  if  further  selection  corrections  were  needed. 

Selection  bias  arises  when  individuals  decide  (self-select)  whether  to  participate  in  a  program  and 
when  some  of  the  factors  that  influence  their  decision  also  influence  the  outcomes  to  be  evaluated.  In 
the  CNO,  it  appeared  that  beneficiaries  who  chose  to  participate  consistently  used  more  services  than 
the  average  beneficiary  in  the  CNO  service  area.  Failure  to  account  for  this  difference  would  confuse 
the  selection  effect  with  the  treatment  effect,  overstating  the  latter. 

Selection  bias  can  arise  because  of  observable  factors  (like  age,  sex,  and  prior  diagnoses)  and 
unobservable  factors  (like  attitudes  and  lifestyle).  Equations  [3]  and  [4]  illustrate  this  distinction: 

[3]      P*it  =  XitY  +  Sit  (P=l  if  P*>0,P=0  otherwise) 

[4]      Yit  =  a  +  Xitp, +Pi,p2+Vit 

where  P  indicates  participation  in  the  demonstration,  Y  denotes  some  outcome  of  interest,  and  i  and  t 
index  individuals  and  time,  respectively.  X  represents  observable  factors  that  influence  both  the 
decision  to  participate  and  the  outcome;  eit  and  vit  denote  unobservable  factors  that  influence 
participation  and  the  outcome,  respectively;  and  a,  p,  and  y  are  parameters  to  be  estimated. 

If  there  are  neither  observable  nor  unobservable  factors  that  influence  both  selection  and  the  outcome 
(p,  =  0),  then  a  direct  comparison  of  mean  values  for  Y  will  produce  an  unbiased  estimate  of  the 
effect  of  the  demonstration  (p2).  This  is  the  case  with  randomized  design,  since  the  random  allocation 
of  applicants  to  either  treatment  or  control  ensures  that  both  groups  are  equal  with  respect  to  both 
unobservable  and  observable  characteristics. 

Comparisons  of  Risk-Adjusted  Means 

If  there  are  observable  factors  but  not  unobservable  factors  that  influence  both  selection  and  the 
outcome,  eit  and  vit  will  be  uncorrected  and  a  fair  comparison  between  participants  and  non- 
participants  can  be  achieved  by  controlling  for  observable  factors.  This  can  be  done  by  estimating 
Equation  [4]  alone,  with  the  estimated  coefficient  on  the  program  participation  variable  (Pit)  providing 
an  unbiased  measure  of  the  treatment  effect.  Our  first  set  of  models  was  based  on  this  assumption.  In 
these  models,  Xit  was  a  risk  score,  defined  as  total  expenditures  predicted  by  demographic 
characteristics,  prior  expenditures,  and  prior  diagnoses. 

These  types  of  models,  commonly  referred  to  as  risk  adjustment  models,  use  information  about 
individual  beneficiaries  from  basic  demographics  like  age  and  sex  to  complex  data  on  prior  service 
utilization  and  diagnoses  to  adjust  for  differences  in  their  baseline  health  status.  It  should  be  kept  in 
mind,  however,  that  it  is  almost  impossible  to  comprehensively  account  for  differences  in  overall 
health  status  by  a  statistical  model.  Risk  adjustment  models  can  therefore  only  serve  the  narrow 
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purpose  of  explaining  differential  risk  for  a  particular  outcome  of  interest.  Against  this  background, 
we  explain  the  construction  of  our  risk  adjustment  model  here  using  total  expenditures  as  an  example. 
The  described  procedure  was  used  in  an  analogous  fashion  for  all  other  outcomes  analyzed  in  this 
report. 

The  overarching  strategy  for  risk  adjustment  was  to  predict  total  expenditures  with  a  very  rich  model 
separately  and  then  use  the  predicted  value  from  this  first  model  as  a  single  risk  score  variable  in  the 
regression  models  that  tested  the  treatment  effect  (Needleman  et  al.,  2001).  In  other  words,  the 
treatment  effect  models  investigated  to  what  degree  CNO-participation  could  explain  the  deviation  of 
the  actually  observed  expenditures  from  the  estimated  expenditures,  which  were  predicted  on  the 
basis  of  beneficiary  characteristics.  Separately  estimating  these  models  streamlined  the  computations 
for  this  project,  since  a  large  number  of  risk  adjustment  variables  could  be  narrowed  down  to  one 
single  score,  increasing  both  speed  of  estimation  and  interpretability  of  output.  In  addition,  the  risk 
adjustment  model  had  to  be  run  only  once  per  outcome,  not  repeated  for  the  different  tested 
specifications  of  the  treatment  effect.  Separate  risk  adjustment  models  were  run  for  each  CNO  site 
and  each  year. 

The  goal  for  the  risk  adjustment  model  was  to  predict  as  much  variance  in  expenditures  as  possible. 
We  therefore  estimated  a  very  comprehensive  model  with  information  on  demographic 
characteristics,  past  utilization  and  past  diagnoses.  Demographic  characteristics  were  captured  by  a 
variable  for  sex  and  four  variables  for  age  groups  (<65,  65-74,  75-84,  >85  years  of  age).  Past 
utilization  was  captured  by  including  the  expenditures  incurred  during  the  previous  year.  It  is  both 
plausible  and  well-established  by  prior  research  that  such  prior  utilization  measures  are  highly 
predictive  of  future  resource  use  (van  Vliet  1993;  Pope  1998),  rendering  them  ideal  candidates  for  the 
proposed  adjustment  model.  They  are,  however,  frequently  not  used  in  a  regulatory  or  rate-setting 
context,  since  basing  future  payment  rates  on  past  utilization  creates  incentives  for  maintaining  high 
utilization  levels.  In  addition,  we  included  squared  prior  expenditures  to  account  for  possible  non- 
linear relationships  between  past  and  current  spending. 

To  capture  past  diagnoses,  we  included  1 1 8  indicator  variables  for  the  prospective  Hierarchical  Co- 
existing Conditions  (HCCs).15  HCCs  have  been  created  to  calculate  risk  adjusted  capitation  payments 
for  Medicare  beneficiaries  enrolling  in  HMOs  (Ellis  1996).  They  are  groupings  of  ICD-9-CM 
diagnostic  codes  into  non-mutually  exclusive  categories  based  on  clinical  logic  that  have  been  shown 
to  be  good  predictors  of  utilization  of  medical  care.  This  holds  particularly  true  for  the  so-called 
concurrent  models  in  which  clinical  diagnoses  in  a  given  time  period  are  used  to  predict  spending  in 
the  same  time  period.  Obviously,  it  is  much  more  difficult  to  use  HCC  models  to  predict  future 
resource  use,  since  acute  care  episodes  account  for  a  large  part  of  medical  spending. 

Despite  the  superior  predictive  power  of  concurrent  HCC  models,  however,  the  prospective  HCC 
model  appeared  more  appropriate  for  this  evaluation.  Consider  the  following  example:  if  enrollment 
in  the  CNO  were  so  effective  that  typical  enrollee  health  status  were  improved  (i.e.  they  had  less 
severe  diagnoses),  their  actual  expenditures  would  be  reduced,  and  the  concurrent  HCC  model  would 
correctly  predict  those  lower  expenditures.  Consequently,  the  difference  between  their  actual  and 
predicted  expenditures  would  be  small.  As  our  test  for  the  treatment  effect  relies  on  a  comparison 


Ellis  et.  al.  (1996)  found  that  concurrent  HCC  models  could  predict  between  41  and  55  percent  of  the 
variance  in  total  expenditures,  depending  on  which  HCC  variant  is  used.  Prospective  models,  however, 
were  much  less  powerful,  predicting  approximately  8  percent  of  variance. 
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between  actual  and  predicted  expenditures,  we  would  incorrectly  conclude  little  or  no  effect  of  the 
demonstration.  Thus,  the  use  of  concurrent  HCC  models  might  unfairly  bias  the  analysis  against 
finding  an  effect  of  CNO  participation. 

In  addition  to  the  main  effects  of  the  described  variables,  we  included  interaction  terms  between  the 
demographic  variables  (age,  race,  and  sex)  and  the  HCCs  variables.  Those  interaction  terms  represent 
the  assumption  that  the  effect  of  prior  illnesses  on  total  expenditures  differs  by  age  and  sex.  Thus,  the 
risk  adjustment  model  takes  advantage  of  the  full  complexity  of  beneficiary-level  information  that 
was  available  to  the  project  team. 

The  resulting  model  is  depicted  in  equation  [5]: 

[5]  Yit  =ct0  +Ageita,  +SeXja2  +Raceja3  +  HCCit_,a4  +  lnteractionsa5  +Yit_1a6  +Y^_,a7  +e 

where  the  subscripts  i  and  t  denote  the  beneficiary  and  the  year,  respectively.  Age  denotes  the  age 
group  vector,  Sex  the  sex  variable,  Race  the  white/nonwhite  indicator,  HCC  the  vector  of  HCC 
variables,  Interactions  the  vector  of  the  interaction  terms,  and  Y  the  expenditures;  do  through  ct7 
represent  vectors  of  parameters  to  be  estimated.  This  model  was  estimated  separately  for  each  site  and 

A 

year,  then  predicted  values  ( Y )  were  saved  and  used  as  the  risk  score  in  our  treatment  models  (Xit  in 
Equation  4). 

The  predictive  power  of  the  risk  adjustment  model  is  illustrated  in  Table  A.l,  which  reports  the 
predictive  power  for  the  full  set  of  models  estimated  for  Carle  Clinic  (results  were  similar  for  the 
other  sites).  As  expected,  the  complex  model  performed  better  than  a  model  based  on  past  diagnoses 
alone,  as  it  was  able  to  explain,  on  average,  11%  of  the  variance  in  total  spending,  whereas  the 
prospective  HCC  models  explain  only  around  8%.  The  explanatory  power  varied  considerably  for  the 
different  measures.  Items  that  tend  to  be  recurring,  such  as  home  health  expenditures  or  physician 
office  visits,  can  be  predicted  much  more  precisely  than  items  that  tend  to  have  a  strong  random 
component,  such  as  hospital  care  or  ER  visits. 

In  summary,  our  approach  to  risk  adjustment  consisted  of  the  estimation  of  predicted  expenditures  on 
the  basis  of  beneficiary-level  information  condensed  into  a  risk  score.  While  the  approach  could  be 
called  agnostic,  since  it  did  not  try  to  uncover  the  contribution  of  each  risk  score  component,  the 
method  was  highly  economical  and  suitable  for  the  particular  purpose  of  this  project.  In  addition, 
since  the  large  sample  size  of  the  population  reference  group  was  already  fully  exploited  when  the 
risk  score  was  created,  we  were  able  to  estimate  the  treatment  effect  by  comparing  the  full  treatment 
group  to  a  10%  random  sample  of  the  population  reference  group,  substantially  reducing 
computational  demands. 
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Table  A.1 


Predictive  Power  of  the  Risk  Adjustment  Model  for  Expenditure  and  Utilization  Measures: 
Carle  Clinic 


1994 

1995 

1996 

1997 

1998 

1999 

Average 

Total  Expenditures 

12% 

12% 

11% 

9% 

13% 

11% 

11% 

In-bundle  Expenditures 

36% 

38% 

40% 

55% 

44% 

47% 

43% 

Out-of-bundle  Expenditures 

11% 

10% 

10% 

8% 

11% 

10% 

10% 

Hospital  Expenditures 

8% 

8% 

7% 

5% 

7% 

7% 

7% 

ER  Expenditures 

8% 

7% 

6% 

6% 

8% 

10% 

7% 

Office  Visit  Expenditures 

34% 

34% 

34% 

34% 

37% 

38% 

35% 

Admissions  [probability] 

11% 

10% 

10% 

10% 

10% 

10% 

10% 

ER  Visits  [probability] 

8% 

8% 

9% 

8% 

8% 

9% 

8% 

Office  Visits  [count] 

41% 

42% 

43% 

42% 

44% 

45% 

43% 

Numbers  express  percent  variance  explained  by  the  model,  measured  by  the  Pseudo-R2  for  predictions  of  probability  of 
admission  or  ER  visit,  and  from  the  R2  for  all  other  measures. 


Sources:    Abt  Associates  Inc.  CNO  Evaluation  Analytic  Files 


Difference-in-Differences 

The  risk  adjustment  models  described  above  are  sufficient  if  selection  is  influenced  by  purely 
observable  factors.  However,  if  there  are  both  observable  and  unobservable  factors  that  influence 
both  selection  and  the  outcome,  then  estimating  Equation  [4]  alone  will  produce  biased  estimates  of 
the  demonstration  effect  (p2)  because  some  of  the  unobservable  factors  that  drive  selection  will  also 
cause  Y  to  be  systematically  high  (or  low)  for  participants.  For  example,  if  individuals  with 
particularly  active  lifestyles  were  recruited  to  the  CNO,  then  participants  will  tend  to  have  lower 
Medicare  expenditures  than  non-participants,  even  if  the  CNO  itself  had  no  effect  on  their 
expenditures. 

If  these  unobservable  characteristics  of  beneficiaries  that  influence  their  decisions  to  participate  in  the 
demonstration  and  also  affect  their  outcomes  (expenditures  and  utilization)  are  invariant  over  time, 
then  the  difference-in-differences  approach  can  remove  the  resulting  selection  bias.  To  illustrate,  we 
can  rewrite  Equations  [4]  and  [5]  to  decompose  the  unobservable  terms  into  permanent  and  transitory 
components: 

[6]      P*it  =  XftY  +  5j  +  kt  +  Eit         (P=l  if  P*>0,  P=0  otherwise) 
[7]      Yit  =  a  +  Xup  i  +  Pi,p2  +  5i  +  I,  +  vi( 

where  5j  denotes  permanent,  unobservable  characteristics  of  beneficiaries  (like  family  history)  and  A., 
denotes  unobservable  factors  that  affect  all  beneficiaries  in  the  same  month  (like  the  weather).  Since 


Abt  Associates  Inc. 


Phase  II  Evaluation  of  CNO  Demonstration 


61 


Sj  and  Xt  are  unobservable  and  influence  both  selection  and  the  outcome,  they  will  result  in  selection 
bias  unless  they  can  be  removed  from  the  estimating  equation  through  algebraic  manipulation.  The 
difference  in  differences  estimator  achieves  this  by  comparing  the  change  in  average  Y  over  time  for 
one  group  with  the  change  in  average  Y  over  time  for  another.  The  first  difference  (the  change  in 
average  Y  for  one  group)  can  be  written  as  follows,  since  the  permanent  components  cancel  out: 

[8]         Y,  - Y_  =  (X,  -Xt_,)P,  +(Pt  -Pt_,)02  +(*.,  -XM)+(vt-vM) 

where  Pit-Pit_,  is  equal  to  zero  for  the  population  reference  group  and  one  for  the  treatment  group 
(provided  t-1  refers  to  a  period  before  the  treatment  effect  is  expected  to  be  observed).  If  the 
expected  values  of  v,  are  always  zero,  then  the  second  difference  (between  groups)  will  cause  these 
terms  as  well  as  (Xt  -Ki)  to  cancel  out.  What  remains  indicates  that  the  difference  in  differences  for 
the  outcome  is  the  sum  of  the  difference  in  differences  for  the  predicted  outcome  (based  on  the 
estimated  risk  score  and  denoted  by  Xp,)  and  the  demonstration  effect  (p2).  A  comparison  of 
differences  in  differences  for  actual  and  predicted  outcomes  therefore  isolates  the  demonstration 
effect  in  an  unbiased  fashion. 
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Appendix  B:  Sample  Sizes  for  Chapter  3  and 
Derivation  of  Contamination  Bias 


Appendix  Table  B.l 

Sample  Sizes  for  Tables  in  Chapter  3  

Number  of  person-months  in  Tables  3.1,  3.3,  and  3.4 

Carle  Clinic-IL     Carondelet-AZ  LAH-MN  VNS-NY 

All  randomized 

beneficiaries                 T         120,093             115,654  74,887  58,412 

C         56,204               56,423  37,309  29,126 

CNO  enrollees  vs  un- 

enrolled  controls            T         99,865               93,726  65,499  47,244 

C         56,204              56,423  37,309  29,126 


Number  of  person-months  in  Table  3.2 

Carle  Clinic-IL    Carondelet-AZ       LAH-MN  VNS-NY 


All  randomized  benefi- 
ciaries 


T         103,466  100,527  66,652  50,967 

C         48,393  48,813  33,222  25,513 


CNO  enrollees  vs  un- 

enrolled  controls  T         86,882  80,938  59,257  41,512 

C         48,393  48,813  33,222  25,513 


Number  of  person-months  in  Table  3.5 

Carle  Clinic-IL     Carondelet-AZ        LAH-MN  VNS-NY 

Nondropouts  65,996  53,740  55,687  27,062 
Dropouts   2J25  2^31  U591  1,685 

Note:       Sample  sizes  for  Table  3.5  exclude  beneficiaries  who  died  or  enrolled  in  the  CNO  intermittently. 

Sources:  Abt  Associates  Inc.  CNO  Evaluation  Analytic  Files,  January  1994-September  1995  and  January  1997-October 
1997 
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Enrollment  of  Controls  in  the  CNO  Biases  the  Estimate  of  CNO 
Effects  Toward  Zero 

Let  xT  and  xc  denote  mean  Medicare  expenditures  for  the  treatment  and  control  groups.  Under 
random  assignment,  the  expected  value  of  xc  is  just  the  population  mean  value  of  expenditures,  u ; 
the  expected  value  of  xT  is  equal  to  the  population  mean  plus  a  treatment  effect,  A ,  which  may  be 
positive  or  negative. 

(1)  £(xc)  =  u;    £(xr)  =  *i  +  A 

Thus  the  difference  between  mean  expenditures  in  the  treatment  and  control  groups  is  an  unbiased 
estimator  of  the  CNO  (treatment)  effect. 

(2)  E(xT-xc)  =  A 

Now  suppose  that  some  proportion,  pc ,  of  individuals  in  the  control  group  actually  enroll  in  the 
CNO  and  so  receive  the  treatment.  The  control  group  is  now  contaminated  and  the  expected  value  of 
xT  is  not  p. ,  but  rather  \i  +  pc  A .  When  controls  are  contaminated  in  this  way,  equation  (2)  does  not 
hold.  Rather 

(3)  E(xT  -xc)  =  (l-pc)A 

Since  0  <  1  -  pc  <  1  the  treatment-control  difference  in  Medicare  expenditures  will  tend  to  understate 
the  true  CNO  effects,  whether  positive  or  negative.  That  is,  our  estimator  of  CNO  effect  is  biased 
toward  zero  unless  pc  =  0 . 

Because  the  number  of  beneficiaries  in  the  control  group  who  enrolled  in  the  treatment  group  after 
October,  1997  is  known,  we  can  estimate  pc .  If  there  are  nc  members  of  the  control  group  and  if 
nc  of  these  enrolled  in  the  CNO,  then  nc  I  nc  is  an  unbiased  estimator  of  pc .  One  might  then 
construct  an  "adjusted"  difference  in  means  as  a  means  of  correcting  for  contamination  of  the  control 
group. 

This  estimator  simply  inflates  the  treatment-control  difference  by  the  reciprocal  of  the  proportion  of 
controls  who  remain  uncontaminated.  The  resulting  estimator  is  not  unbiased,  but  is  consistent. 
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Appendix  C:  Sample  Sizes  for  Chapter  4 


Table  C.1 

Description  of  the  Samples  for  the  Regression  Based  Analyses:  Number  of  Person-Years 
Analyzed  for  Different  Definitions  of  the  Treatment  Group  

Carle     Carondelet  LAH  VNS  Total 


Randomized  to  treatment  group  (Tables  4.1.1,  4.2.1,  4.3.1,  4.3.2,  4.3.3) 

T                      17,018           17,673            9,042             9,920  53,653 

P                      41,572           37,325           49,644            57,188  185,729 

Total                  58,590           54,998           58,686            67,108  239,382 

Randomized  to  treatment  group  and  enrolled  for  at  least  six  months  (Table  4.2.2) 

T                       11,740            11,373             6,811              6,339  36,263 

P                      41,572           37,325           49,644            57,188  185,729 

Total                  53,312           48,698           56,455             63,527  221,992 


T  denotes  treatment  group,  P  population  reference  group. 
Sources:    Abt  Associates  Inc.  CNO  Evaluation  Analytic  Files,  January  1 994-December  1999 
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